Industrial-scale Web Scraping with AI & Proxy Networks

[@beyondfireship]

Use this link to get a $10 credit, enough cash to scrape thousands of pages get.brightdata.com/fireship

[@ItsDeanDavis]

❤

[@Reddblue]

This man selling wood and iron to shovel makers

[@anze]

@beyondfireship ad link doesnt work

[@NoahKalson]

​@@anze worked for me. Try now.

[@tamasmajer]

The pricing page says 20$/GB. I checked how big the pricing page was it loaded 4MB, so then it costs 20$ for 250 pages? That seems very expensive. Or how should i calculate the price?

[@rvft]

I like how he didn't use "cheap" during the entire video because my god the pricing is absolutely madness on the advertised product

[@brunopanizzi]

Industrial scale!!!

[@koba2160]

scraping aint cheap, but theres many ways to make it much cheaper

[@mrgyani]

​@@arteuspw what do you mean by 1gb/$1? You mean browsing 1gb of data for a dollar with a single proxy? How many proxies do you get for $1?

[@ИмяФамилия-ч3и6щ]

@@arteuspw Please tell me where to buy them at this price.

[@mantas9827]

Is 20$ per GB considered expensive? I wonder how much could you scrape from a site like amazon for that GB... surely a lot ?

[@albiceleste101]

As a freelance dev I get contacted all the time for scraping, it's definitely one of the most requested along with Wordpress (which I also dont work with)

[@cymaked]

interesting - 8 years of freelancing and never had one such request 😮

[@dinoscheidt]

And with a freelancer, the business has the advantage that YOU break the terms and conditions of the companies you scrape (are legally liable and suable). Not the business 😊 so a cheap code monkey and legal scape goat all in one 💪

[@mrgyani]

Where do you get these projects from?

[@VividCoding]

@@dinoscheidt Wait can they really do that? They are the ones who wanted to scrape the data in the first place.

[@dabbopabblo]

I'm not even a freelancer and I cant count the number of times on two hands Ive been asked to make someone a website. They think because I'm a web developer I am just some guy who goes around making websites willy nilly. And the few times I have actually went through with helping someone out, they want everything Wix or Wordpress provides and have the audacity to suggest I shouldn't be asking so much in pay when a drag and drop builder can suffice.. THEN USE THE BUILDER GOD DAMMIT. My knowledge is wasted on front end work anyways.

[@alexcasillas2488]

This reminds me of when I solved 100 captchas manually so that I could download some data files from a website for an ai. I got a sever message temporarily banning me from the website saying that I must be a bot. I learned my lesson and stuck to only solving 99 captchas each day from then on until I had enough data files

[@Loubensdoriscar]

Zeus Proxy's specific emphasis on session management is a key factor that resonates with my goal of executing data retrieval tasks with a focus on mimicking genuine user behaviors.

[@yashkhd1100]

To be frank out of all youtubers Fireship has most interesting and to the point videos and gives most value out of time spend. Kind of just wondering how he keeps track of all the varied topics and able to make most out of it.

[@julienwickramatunga7338]

He already has five prototypes of Neuralink chips plugged into his brain, linked to the Web via 5G, and he is using digital clones of himself (coded in JS of course) to make more video content (with the help of ChatGPT). That makes him the most powerful being on the planet. Praise the Cyber-Jeff! 👾

[@AdamBechtol]

Mmm

[@clerpington_the_fifth]

well thankfully there's this thing called a search button

[@EliteGamerpk]

As a web scraping tool developer, one thing to note about the chatGPT code about extracting product names etc is that it's not going to work on all cases. What I mean by that is we can see there are some random class names like '._cDEzb'. And these classes can vary from page to page. So your code for one listing page, might not work for other. The way I do this is using some advanced query selectors that don't rely on unreliable classes. Can go into more detail if required.

[@CrackedPlayz]

Please do!

[@RiChYFanatics]

Dont be shy :p

[@myhitltd5826]

so that's why I copy full selector of the element and work with it in puppeteer.

[@MrNsaysHi]

AFAIK puppeteer doesn't support finding elements by xpath, so what do guys use?

[@thrand]

@@MrNsaysHi well, real men write their own html parser and query language. But peasants like myself use css selectors with document.querySelectorAll.

[@wtfdoiputhere]

Web scraping is still my favourite type of projects it's so fun and "meaningful" to me and with the help of AI i can see it becoming much much easier

[@0187]

same, gives me shitton of satisfaction

[@GeekProdigyGuy]

thanks Jesus

[@alejandroarango8227]

Unfortunately GPT4 is still too expensive to use in projects and gpt3.5 is still too stupid.

[@wtfdoiputhere]

@@alejandroarango8227 it's stupid enough so you still do much of the work yourself cause eventually it's just a tool to help and personally it helps me enough

[@wtfdoiputhere]

@@0187 exactly what i feel

[@YuriG03042]

toward the end of the video, Jeff suggests that you can grab all the links and then make requests to those links. it gave me flashbacks of another video on the main channel where a company did this and ended up with a 70k+ GCP bill after one night of web scraping, because their computing instance was forever recursing and was scalable up to 1000 instances lmao

[@xanderbarkhatov]

If I'm not mistaken, page.waitForSelector(selector) already returns the element handle, so you don't need to use page.$(selector) after that. Anyway, great video, as always. Thank you! ❤

[@yvanguemkam4739]

You're right, wanted to said that... But don't have money to spend on the browser. Is there an alternative?

[@cyberzjeh]

​@@yvanguemkam4739 you can host puppeteer yourself and pay for a proxy service if you need it, might come out cheaper (but more work obviously)

[@meansnada]

I love how there are legit businesses to bypass captchas and mess up with data :)

[@dislike__button]

Scraping isn't illegal

[@Tylersmodding]

and individuals

[@aresakmalcus6578]

@@dislike__button if it's against Terms of Service of the given site, it is

[@Bruceylancer]

@@aresakmalcus6578 I'm not a lawyer, but how can it possibly be illegal? It can be against ToS, sure, then the website owners can surely act accordingly, i.e. ban your account on the said website, ban your IP address, and so on. But illegal? Are there any laws out there that prohibit collecting public data? Are there any cases of people getting sued for scraping? I haven't heard of such, maybe you can provide some examples. Also, there are 8-figure businesses built on scraping, like Ahrefs or Semrush.

[@Bruceylancer]

@@Andrew-zy7jz Exactly! Very good example.

[@Wei-KuoLi]

Thank you for teaching me puppeteer and bright data, beats all content on internet

[@abz4852]

fireship you are uploading videos faster than new javascript frameworks get released

[@Maneki-Nico]

Your videos are somehow exactly relevant to the code I am writing every week - interesting for sure!

[@Jeanseb23]

You've foiled my plan 5 years in the making. At least now I have a free 10$ credit for Brightdata to catch up. Thanks Fireship!

[@gatonegro187]

how much did u end up spending

[@shawnvirdree8593]

Wow, you’re on the cutting edge of technology 🤯

[@BharadwajGiridhar]

One thing jeff is that these websites change css class names on every refresh. So it's better to write code with selectors that don't change like id or aria label.

[@Ruf4eg]

Man, you are reading my thoughts! this video came at the right time when I wanted to scrape some websites!!!!

[@ikedacripps]

When I first saw puppeteer when I was learning nodejs this is exactly the kind of use case I wanted to apply it to. Specifically wanted to scrape csv files and have some AI learn it and make some sense out of it. I think it’s now more than possible

[@DemPilafian]

Downloading CSV files would typically not be considered _"scraping"._ You don't have to scrape the data out of a CSV file -- it's already data.

[@ikedacripps]

@@DemPilafian you just wanna falsify my statement but scraping for csv file is as valid as scraping for pdf files. I specifically wanted to scrape soccer analytics websites for those csv files. Hope that puts it into perspective for you .

[@DanielLavedoniodeLima_DLL]

I remembered that web scrapping was a nightmare to deal with, specially doing this proxy rotation by ourselves. This tool is not cheap, though, so at least here in Brazil (and other emerging countries alike), companies will still be doing that like the old days. The captcha solving was actually done by real people at the time I worked in a company that mined those kind of data a few years ago, but I guess this can be automated with GPT-4 tools now

[@aseluxestays]

I'm here because I need to hire someone who can provide this service for me. Great video!

[@TheHassoun9]

Hi I'm willing to help# I'm a dev looking for commission

[@abishekbaiju1705]

Thanks for making this video. I am actually working on a project where the users can add amazon products and look for price changes and also get notified with price changes. My objective was to learn web scraping.

[@beefykenny]

This video has a lot of value.

[@senorclouds]

This is a great spell for Howarts Ai Academy, Thanks Professor Fireship ^^

[@NathanDodson]

See. This is why I watch all your videos, Jeff. I'm a super shit JS coder, but I'm pretty decent with Python. This gives me an idea for my own eBay business, and scouring those tool docs for Python SDKs to do the same thing. Honestly, it's been your videos that have kept me in the coding space. You always have these creative "concept/idea" videos and a good majority of them have me opening up VSC to do some tinkering. Thanks for all your content brother.

[@priapulida]

there's Pyppeteer

[@maskettaman1488]

@BeBop No, it's Pyppeteer

[@minhuang8848]

@@bebop355 *pyppeteer tho

[@JGBreton]

did this materialize?

[@tonymudau3005]

​@@JGBretonlmao 😂 asking myself the same thing

[@nichtolarchotolok]

Been using puppeteer for a few yrs for freelance web scraping. Puppeteer and Playwright have been a saving grace in many circumstances.

[@donirahmatiana8675]

could you give some tips to not getting ip banned?

[@nichtolarchotolok]

@@donirahmatiana8675 puppeteer-extra library and the puppeteer-extra-stealth plugin. If that doesnt work, you'd need rotating proxy like that of bright data as mentioned in the video.

[@jacekpaczos3012]

@@nichtolarchotolok are you not using scrapy? I always thought of scrapy as the most convenient solution.

[@nichtolarchotolok]

@@jacekpaczos3012 I started off on the nodejs route and havent had the need to try the python way of doing this. I do remember trying scrapy in my early days but for some reason puppeteer felt more intuitive to me. That is probably because I felt more comfortable writing javascript code.

[@kellymcdonald7095]

I just saw a comment above saying clients request a web scrapping tool but if it's not legal to scrape the website then how do you take up freelance web scraping what if the guy uses the data and the company ur scrapping from finds out about it? will you not be in trouble or how does this work ?

[@prabhavkhera4959]

Thanks Jeff. I was planning on building a project that uses web scraping and this video absolutely dropped at the perfect time. Appreciate it. I love your videos and hope for more such content in the future :)

[@forbiddenera]

Puppeteer is the source of non stop memory leak nightmares for me. Fortunately I got it down to under like 30mb a day but originally it was like 30mb per leak and like 250+mb a day leaked (and it was mostly only loading 2 pages back and forth)

[@alejandroarango8227]

I avoid using it to the maximum, it is a waste of server resources.

[@andy12379]

You could just close the browser and open a new one every time you use it to avoid memory leaks

[@wlockuz4467]

Remote browser as a service is actually a genius idea. Often times when you want to scrape at scale the most painful thing to do is hosting and using effective proxies. But with this you can literally leave the scraper running on your machine and let brightdata take care of the proxies. You don't even need good specs because the browser runs on a different server.

[@quickkcare605]

Well thought!

[@klapaucius515]

smells like ad

[@wlockuz4467]

@@klapaucius515 Do you mean that for my comment or the video?

[@arrvee7249]

ikr, then you can just pay brightdata $10,000 and go on to make $52 for the data you've scraped.

[@blaizeW]

Another gold gem for daddy fireship 🤑🔥

[@danieldosen5260]

I never thought of returning data as JSON... that's obvious and brilliant...

[@selimachour]

I usually block the fetching of images, css, fonts (and javascript if the website can run without) which speeds up the page load by a lot!

[@felixmildon690]

Best video yet thanks fireship. This will introduce me to puppeteer and the services BrightData offers (BrightDatas prices are a concern based on the comments section)

[@CODE_YOUR_TYPE]

I love you man i was trying for so long and you are the only one who gave the solution thank you so much

[@d3layd]

Thank you for this! I used ChatGPT to write a puppeteer script for me the other day and it was fucking slick

[@classmanOfficial]

Selenium has a headless mode :) if you guys want to try it out, works well enough for multithreading

[@kevinbatdorf]

some of those query selectors look like they’d break in a week. Maybe you need to add openai to the workflow more directly

[@RichardHarlos]

It's a proof of concept/tutorial, not an explicit recommendation for bulletproof boilerplate. Context, eh? :)

[@yellowboat8773]

Maybe outputting the html every time to openai then having that pick the query selector then insert into the script. Do have to be very specific with your prompt because it often replies with: The query selector is: a.carousel

[@tioModz-w6i]

Спасибо автору за новую связку. Проверил, всё работает.

[@felixmildon690]

Tutorial starts at 2:15

[@AnshTiwari-fx2yq]

May god bless you

[@Victor4X]

Stuff isn't censored properly at 3:00 But I assume those creds are temporary anyway

[@cymaked]

theres many videos on Fireship where he jokes about living dangerously and letting the cred be seen 😂 obv temp stuff

[@thie9781]

​@@cymaked or just F12 to let somebody waste their time

[@EuricoAbel]

Incorporating Zeus Proxy into your SEO strategy ensures efficient and effective monitoring and data gathering processes.

[@bossdaily5575]

Virgin API users vs Chad Web scrapers

[@clerpington_the_fifth]

The Chad's never read the TOS, LoL

[@Autoscraping]

An extraordinary piece of video material that has proven highly useful for our new team members. Your generosity is immensely appreciated!

[@VaibhavShewale]

damn that was really amazing, i was actually thinking of taking snippet of the page extract data then delte that page and repeat

[@JoãoVitorDeSouzaSouza-b2e]

Отличный вариант, попробовал, все работает, спасибо! ❤

[@chaseclingman]

I liked how you showed the timeout as 2 * 60 * 1000 so beginner friendly haha

[@mrgalaxy396]

I mean that's way more readable than 1200000, this is a pretty common practice

[@clerpington_the_fifth]

@@mrgalaxy396 could also do 1_200_000 if it was python

[@hamza-325]

I worked for a digital shelf company that scrap the data from Amazon and more websites. They use many proxy services but one of the most expensive ones was BrightData, so the more experienced workers always instructed us to not use BrightData unless it is really necessary.

[@sciencenerd8326]

what are the others that are better?

[@hamza-325]

@@sciencenerd8326 the company has made some cheap proxies using the machines of AWS for examples (they don't have many IPs but they do the job for many websites). And I think there are cheaper services like ProxyRack.

[@fhnvcghj1587]

​@@hamza-325I have a task of selenium bot I have 1000 account but need 1 ip for each account to make request to the website and do the work any idea or paid service for that

[@maxivy]

Awesome video - I will have to rewrite it in Python though ;) because I am a human bean

[@NicolaiWeitkemper]

BeautifulSoup is better anyways :P

[@priapulida]

@@danielsan901998 or Pyppeteer

[@NicolaiWeitkemper]

@@danielsan901998 Correct, that's not an even comparison. However: BeautifulSoup >> Cheerio

[@estebancordoba555]

In my country, some products are more expensive than amazon, I built a scrapper to get the products and price with params as the brand or names but amazon blocked me couple of times, this si really nice solution!

[@boba-b5n]

Огромное спасибо за рабочую связку.

[@KabbalahredemptionBlogspot]

OK that was way cooler than I thought

[@kinglane8634]

Thanks for always helping us devs keep out workflow clean and simple!!! If you plan on starting a subscription service I'd love to see what you're offering.

[@trickster6254]

He has got a website offering courses. I bought the Angular one myself and was really good.

[@ehsanpo]

web scraping with ruby and rails is one of the best ways

[@Kevgas]

You should create a course on how to do this, Id pay for that!

[@hermanplatou]

Doesn't amazon rotate the classes and ids, effectively breaking your selectors? Not sure how the most advanced RPA bots work, but im hoping that some of them offer a AI that grabs screenshots and parses them instead. Would be interesting with a follow up!

[@makkusu3866]

Yea, I think classes should be autogenerated, at least after every deployment if not every request. Fast and dirty solution would be to use openapi sdk to prompt ChatGPT to generate document query code and eval it

[@trappedcat3615]

@@makkusu3866 You can select elements based on attributes or lack of attributes, or you can use pseudo-classes such as :nth-of-type. There are dozens of them.

[@iljazero]

@@trappedcat3615 yea, that is how i wrote scraping for other website, i targeted div elements with style X which often ... doesn't change cuz... why ;D

[@arthur0x2a]

Most websites with random ids/classes names still have a common and repetitive structure. Axios + Regex and you'll process ~10 time as much pages as puppeteer, with minimal bandwidth by default and simpler code. Just validate the output with a strict schema (as you always should) and you'll maybe have to update it once a year at most. Puppeteer's only real advantage is TLS fingerprint

[@cyberzjeh]

​@@arthur0x2a you can also use sg like cheerio as a middleground between an entire headless browser, and parsing html with fukken regex (chad move tho ngl)

[@ozten]

Those css selectors look super fragile.

[@RichardHarlos]

It's a proof of concept/tutorial, not an explicit recommendation for bulletproof boilerplate. Context, eh? :)

[@AbuBakar-pc2fp]

Awesome Explanation

[@Jason-nv6ku]

You're amazing! Many thanks!

[@rstar899]

Amazing video as always 🎉

[@Xld3beats]

Guess its time to write a program that applies to every job on the internet

[@maxivy]

This may be the first time I actually was excited about a sponsored segment and will actually sign up for the product

[@MrKrzysiek9991]

Microbots AI chrome extension helps with building prompt with HTML code included. Chech it out it you want to write automation code faster.

[@forbiddenera]

..while Puppeteer can run headless, you don't have to run it headless. It may still seem headless from what most might consider that term to mean but headless or not is a config option for Puppeteer, running with headless disabled can help beat bot detection sometimes.

[@bigplaneenergy8674]

@Beyond Fireship You left the username/passwords unblurred there for a second

[@v1s1v]

Nice tutorial, but there are AI tools now like Kadoa that can do all of this for you. In the time it takes for you to watch this video, you can get an AI scraper up and running.

[@TPAKTOPsp]

Any reason why you have used puppeteer over playwright? I see bright data has support for both.

[@alihansencan394]

Great information thank you !

[@kusztelson2947]

One problem you may face is that class names used for selectors change over time as they are generated every time website is deployed breaking yor code.

[@assmonkey9202]

Reverse engineer algo for generating class names

[@daniamaya]

Gold. Just pure gold.

[@kasparsc]

Sir, you are a legend 🔥🔥🔥

[@CandyLemon36]

I'm impressed by the depth of this material. A book with corresponding themes was a key influence in my life. "AWS Unleashed: Mastering Amazon Web Services for Software Engineers" by Harrison Quill

[@Dev-Siri]

just as I thought the ai videos ended

[@exploringcrypto6609]

Jeff how can you process data so fast?

[@KhaledAlMola]

That is a cool website to use. I'll try it one day

[@TheMalcolm_X]

This video felt like one giant sponsored ad.

[@wandenreich770]

Very insightful

[@harisonfekadu]

You're ingenuity is something else. It's devs like you that won't be replaced by AI.

[@asperthickkgamerr491]

cuz he is an AI

[@manfredcomplex366]

Freaking Money Glitch. Love you man❤

[@nskiran]

We used to user selenium web driver ( webactions) and phantomjs to scrape data. Ip problems were solved with nohodo In good olden days 2014 stack

[@danvilela]

Brooo, this is awesome!

[@luxurycondobbmg]

I remember my first time scraping a website - except back then, we didn't have ChatGPT proompts to do it for us. We had to physically read the documentation and actually understand the code we wrote

[@LukasKlinzing]

Came here for the "AI" and "industrial" for. Got basic scraping with no AI . yay

[@tw-wp5uv]

Bright Data is quite expensive with average success rates for webistes with high protection measurements. keep that in mind if you want to scale your scraping

[@progamer1196]

as soon as I saw the thumbnail I knew this was an ad for brightdata

[@NuncNuncNuncNunc]

5:10 Works until the generated class names change the next time site has a minor update.

[@summonlucifer]

If you use selenium to open a browser window you can easily scrape from any website

[@oblivion_2852]

Could we have a vid on the difference between Selenium and Puppeteer?

[@daniel_q40]

Data is the new gold

[@F1NEk]

you totally can see username and password at 3:01

[@kevinbraga9526]

Great video, i have a question for you, how do you know that this is the industry standard for modern web scraping? Like how can you find out this information.

Only for this topic alone its worth to learn python along with Scrapy

[@JustBR0]

Bright data is throwing their money!!

[@robertwitzke6134]

great video!

[@calmgee]

This was gold

[@SteveHazel]

pretty good "how to be a hacker" intro heheh.

[@avi12]

Nah man, Playwright is the new Puppeteer

[@parlor3115]

This Bright thingy alleviates most of the overhead of web scraping (IP rotation and captcha solving). And goodness GPT-4. Idk if I should feel happy about the potential prospects or anxious about how crazy good this thing has gotten.

[@shaharyarkhan7553]

Can you please make a video on how to efficiently fetching divs with dynamic classes etc

[@keithmofley8275]

I use Selenium at work to scrape my database and output useful excel files rather than spending hours building them by navigating the DB myself.

[@mantas9827]

interesting

[@gior987]

why don't you just use SQL? you have an excel DB?

[@keithmofley8275]

@@gior987 no backend access, can only see the front facing website cause the giy who designed it is both retired and 3rd party so he's not an employee

[@Lars16]

@@keithmofley8275 When you leave the company, some poor guy will be hired to replace you and will have to write some code that takes your Excel file and convert it into whatever format they will need then. And so the cycle of suffering continues.

[@keithmofley8275]

@@Lars16 you know I've been thinking about that a lot so I've made my scripts as easy as possible to understand and edit. There's almost more instruction and explanation than there is code at this point.... He's still going to have to work at it to get what he wants, but hopefully I make it easier on him than the last guy has made it on me. 🤷‍♂️

[@3rawkz]

Scrapy all day baby!