How to scrape ANY website with Apify’s Web Scraper API

[@Apify]

Try out Web Scraper on Apify Store 👉 apify.it/4ebR8RB

[@pushingpandas6479]

How can I find websites and then put the urls into text and search the websites for terms?

[@Apify]

Hi, here's a possible solution: 1. To find websites you can use the Google Search Scraper 2. Upload the resulting URLs of these websites into the Website Content Crawler 3. Search the output for terms manually or using a data analysis tool Hope I helped!

[@edulitix]

Hello, I am a Pro customer with you and have used the scraper. I only get two lines of data from a large page. What am I doing wrong? and with the content crawler I get websites that I don't know and haven't specified. I only wanted to get one website in the content. I received everything else except the content of the website.

[@Apify]

Hey, if you're a customer using one of our paid plans, you can chat with customer support, who'll be able to help you debug the issue for your specific run 🙂 Hope I helped!

[@adetectiveinmedicine7283]

I want to scrape all pages of the website not only the main page of the website, how?

[@metalwarrior6657]

There is a glob input for enqueueing other pages so you can set it up to go on all pages that start with the original domain.

[@Apify]

Exactly! You can read more about glob patterns in the scraper's readme: apify.com/apify/web-scraper#glob-patterns

[@davidoludepo]

Thank you

[@seraquefoisorte1753]

What a terrible example

[@Apify]

Can you elaborate on what was wrong with our example? We're open to any feedback! 🚀