Build undetectable Amazon scraper with n8n, Puppeteer and Scraping Browser

In this tutorial I explain how to automatically scrape Amazon search results using Puppeteer, Scraping Browser and n8n.
In this tutorial I cover:
1. Creating a base Puppeteer script to retrieve data from Amazon search results (product names, prices, ratings etc.)
2. Key issues while scraping Amazon
3. Implementing Scraping Browser into the code
4. Adjusting the script and deploying function to Google Cloud Functions
5. Automating the scraping using n8n (transferring results to Baserow)
Resources:
1. Puppeteer script (for local use): gist.github.com/workfloows/d7...
2. Puppeteer script (for local use with Scraping Browser): gist.github.com/workfloows/1d...
3. Puppeteer script (for use with Scraping Browser on Google Cloud Functions): gist.github.com/workfloows/5a...
4. Scraping Browser API: brightdata.com/products/scrap...
5. Docs for Scraping Browser troubleshooting : help.brightdata.com/hc/en-us/...
My other tutorials:
1. How to build voice Telegram AI bot with n8n, Whisper and ElevenLabs: workfloows.gumroad.com/l/tele...
2. How to build Telegram AI bot with n8n and 🦜🔗 LangChain (FlowiseAI): • How to build Telegram ...
3. Web scraping data with n8n and Puppeteer: • Web scraping data with...
4. How to automate Notion databases using n8n: • How to automate Notion...
5. Using webhooks in n8n (parameters, responses and triggers): • Using webhooks in n8n ...
Subscribe my newsletter: workfloows.com/
Visit my Gumroad profile: workfloows.gumroad.com/
Follow me on Twitter/X: / workfloows
Follow Workfloows on LinkedIn: / workfloows
Disclaimer: I cannot be held responsible for any consequences resulting from the use of the information provided in this tutorial. Make sure to obtain proper authorization before engaging in web scraping activities, or consider using proxies to protect your online presence and ensure ethical scraping practices.
Create your Bright Data account and get $10 credit: brdta.com/workfloows
Create your n8n cloud account here (affiliate): n8ngmbh.partnerlinks.io/6hvl7...
Screen recording software that I use (affiliate): www.screen.studio/@jmMwX
0:00 Final result
1:05 Step 1: Install Puppeteer
1:57 Step 2: Add scraping script
6:48 Step 3: Connect Scraping Browser API
9:14 Step 4: Adjust code to Google Cloud Functions
9:59 Step 5: Deploy function to Google Cloud Functions
12:03 Step 6: Automate scraping with n8n

[@workfloows]

Hello, thanks for watching my video! If you want to play a bit more with scraping using n8n and Puppeteer, here is my previous tutorial: kzbin.info/www/bejne/j6DRf32ndqarmsk

[@jinqimao4781]

thanks for teaching lesson start from the install package. it's so important!!! Keep doing it.

[@workfloows]

Thank you very much - I’m glad you find my video helpful!

[@nestiq]

Congrats bro! 😃😃😃😃😃😃

[@workfloows]

Thanks a lot!

[@SUDHANSHU934]

I didn't like this video....I loved this ❤ There are so many new things which I have learnt from this video today. By the awesome video editing....😍

[@workfloows]

Thank you very much! I'm happy that you like the video and I'm very grateful for your support. All the best to you!

[@RolandoLopezNieto]

Great video

[@workfloows]

Thank you very much!

[@pjones6749]

Great video. Question: I am trying to reduct cloud costs for all these tools. How do you host Baserow, and do you host it on the same server as n8n? I now use Hetzner but not sure if it can handle both.

[@workfloows]

Hey, thank you very much for your comment and kind words about my work - I really appreciate it! Yes, it is possible. Although I haven’t self-hosted Baserow yet (I use cloud option for now), I’m pretty sure it should not be a problem to have both n8n and other apps on the same VPS. The key here is to set each app on a different port. I suppose basic machines with ~2GB memory should handle both things.

[@pjones6749]

@@workfloows Thank you....Subscribed

[@ablae1234]

Is it possible to take this code and modify it to scrape Reddit or any other website, or each site needs a different approach ?

[@workfloows]

Hi, thank you for your comment! This code was created exclusively for Amazon, so scraping and retrieving data from Reddit is rather not possible with it. Of course, you can create your own scraping script using Puppeteer, and I strongly encourage you to do so - it's a lot of fun!

[@Falalfel]

Why do you upload node.js code to Google cloud and not directly in node code in n8n workflow? Is it because you can't import modules? I get module not found error. Is there any others way?

[@GehirnGoldmine]

What is the purpose of this scraped data? Obviously, there seem to be some use-cases. Would you please share some?

[@workfloows]

Hello, thanks for you comment. Absolutely - mostly analytical ones, for example: monitoring of price changes, checking product fit and competitiveness, exploring customers preferences (basing on number of products sold), SEO analysis (e.g. what keywords competitors use) and many more similar. Basically, the key here is not only the type of data scraped, but also automation around it. In this example I scraped search results only for one keyword, but imagine you’d like to perform analysis for dozens or hundreds of product types - it’s also possible with this workflow. Scraped and structured data is also much easier to read and transform - it simplifies performing analytical tasks.

[@GehirnGoldmine]

@@workfloows Hey, Thank you very much for your great answer! Makes a lot of sense to me.

[@chowadagod]

lovely tutorial. i tried this but i'm getting a CORS error.. any idea how to resolve this.. thanks

[@workfloows]

Hi, thanks for your comment and apologize for late feedback. Could you please let me know on which point you get this error (while running script locally or on GCF)? Do you get any other information from console? Thanks in advance for your kind reply.

[@chowadagod]

@@workfloows I later on figured out the error . Thanks 🙏

[@nocodecreative]

what about the puppeteer community node?

[@workfloows]

Hello, thanks a lot for your comment and sorry for my late feedback. I had a chance to use Puppeteer community node, and unfortunately I find it a bit buggy. Since Puppeteer is also slightly demanding in terms of memory, it’s much more convenient for me to host it on GCF and perform calls when needed. But it’s just my experience - if Puppeteer community node works well for you, I don’t see a reason not to use it 😃

[@omare5383]

What do you mean by undetected ? Shouldn't you get blocked after numbers of requests?

[@workfloows]

Hello, thank you for your comment. As long as you use IP rotation, chances that the script will be permanently blocked are rather low (when one IP address is getting blocked, the other one is used in the next round).

[@omare5383]

How can I apply IP Rotation on this method on veido ​@@workfloows

[@wallpp]

aguante el mate

[@workfloows]

¡Aguante!

[@MK-jn9uu]

What’s the purpose of alllll this, if you’re just going to use brightdata

[@workfloows]

Hello, thank you for your comment. In the video description, you can find links to the Puppeteer code without Bright Data implemented. If you prefer not to use BD, please feel free to use these resources and adapt them to the requirements of any other proxy provider of your choice.

[@pratikguptaji]