Ultimate Guide To Web Scraping - Node.js & Python (Puppeteer & Beautiful Soup)

Рет қаралды 33,512

Күн бұрын

Пікірлер: 72

@thewalrusdragon9579 5 ай бұрын

Just wanted to point out how much I appreciate that you take the time to make the audio loud but clear in your vids man. It’s an overlooked thing by many content creators, and it just shows how much you care about the quality of, not just the subject, but the overall experience of your content.

@gochaavsajanishvili1574 5 ай бұрын

Wow I've never been here so early, ty Brad for your tutoring, your videos made me land jobs many times throughout the years! Forever grateful to you!

@eddieantar6958 5 ай бұрын

Damn Brad.. Im one of your Day Oners... offered you that Yankees hat many years ago.. Its so awesome to see you have 2.26M subscribers mans. I think you had that tech guy shop.... I appreciate all your tutorials man but i also valued your off topic videos talking about your struggles in your past and coping with difficulties life throws at you. Thanks for everything you do for the aspiring developer.. Im sure thousands of your views have been gainfully employed by watching your tutorials. Keep it up!

@andybabenko1322 4 ай бұрын

I've been watching you since 2014 - great respect. Good diction and presentation of material. Lack of water and meaningless thoughts. I recommend it to all my students. Thank you

@golden7877 5 ай бұрын

dude its crazy how i was looking for this exact video days ago , and here you upload it , much appreaciated.

@ZenSchafer 3 ай бұрын

this is very good way to teach online. Some ytbers jumps directly into coding part , but you did absolutely good job

@zainulkhan8381 4 ай бұрын

The only tutorial you need for a start. Great work sir.

@abdullahCoder-Tech 5 ай бұрын

I really appreciate your efforts in putting out this content

@kushi1515 4 ай бұрын

That‘s exactly what I have been looking for! Thank you so much for sharing this, helped me a lot.

@pedroernesto5283 5 ай бұрын

If you could make an updated video specifically on Dynamic Websites.... That would be awesome!!!! Most of the valuable data nowadays seems to be on dynamic websites and it`s so hard to deal with them.

@Salah-YT 5 ай бұрын

Thank u so much my best teacher ever

@ts3798 5 ай бұрын

Nice to see you again Brad.

@marianivanov6431 5 ай бұрын

i tried and it works a charm. excellent!. I used zendrows scrapper didnt open the site cause cf. When i try to bright data its work as well great.I will move my all project to bright thank you all

@unclelukedigitals 5 ай бұрын

You're a king Brad. Currently viewing ❤

@Anime_Manias 2 ай бұрын

চমৎকার ভিডিও ভাই ।

@IainEmslie 5 ай бұрын

To setup a Python environment on Windows do this: python -m venv env sourcee env/Scripts/activate

@kurtddanielbigtas1101 4 ай бұрын

Hey Brad, your channel really helped me when I was just starting to develop apps using Angular. There aren't many updated tutorials available for the latest versions of Angular. Could I request a tutorial on Angular PWA?

@sabuein 5 ай бұрын

Thank you, Brad.

@ArtVandelayInc 5 ай бұрын

So good! Thank you

@MichealMicheal-z5d 5 ай бұрын

What about the laravel course you promised to release

@ChocolateCookieMusic 3 ай бұрын

He just announced it in the newest video!

@bharatdubey83 5 ай бұрын

You have revamped your HTML, CSS, Javascript and Next.js courses. When are you planning to revamp your react course?

@ZenSchafer 3 ай бұрын

excellent resource

@oliversieke7071 5 ай бұрын

Hey Brad, I've been following your tutorials and courses for years, and I really love your work! I was wondering if you could create a guide on JavaScript-driven websites. Is it possible to scrape them without using an automated browser? If not, a guide on scraping with a headless browser (without a GUI) would be greatly appreciated. Thanks again, and take care!

@RickBeacham 2 сағат бұрын

Are you referring to using JS from the front end side for scraping other websites? From node (server side): Cheerio is a fast, flexible, and lean implementation of core jQuery designed specifically for the server. It allows you to load and manipulate HTML in a similar way to jQuery. Best for parsing HTML and extracting data. Combining Axios for making HTTP requests and Cheerio for parsing HTML is a common approach for web scraping. Browser based: jQuery, The Fetch API and DOMParser.

@1122slickliverpool 5 ай бұрын

Your timing is scary. ❤😂

@arunkumarsingh8754 4 ай бұрын

Hi will angular 2024 crash course will come ?

@fabrizio.dipietro 5 ай бұрын

Hi, Are you planning a complete corse on udemy?

@abdullahqadi2270 5 ай бұрын

My sincere thanks..

@Ivan-yf2lu 2 ай бұрын

Great video! Can someone tell me if they tried Node.js VS Python for their scrapping and what is performance difference?

@lordp 5 ай бұрын

super useful!

@pratyushdeb5045 4 ай бұрын

which software do you use to make the presentations for the videos and the animations and slides

@adriantepes-qu8wm 4 ай бұрын

Hello Brad, have you done a course on something where you build a project that is like a booking type of app, like car rentals ? Ive looked here and on Udemy but you have so much material that I'm a bit lost.

@buddafi 5 ай бұрын

I guess its summer over there?

@caizer29 5 ай бұрын

What are some legitimate use cases for a web scraper in 2024?

@sulemanahsan7913 5 ай бұрын

please guide us on bypassing captcha and antibot detection.

@RickBeacham 2 сағат бұрын

LLMs, updating your headers/user agent, and proxies would help. Many AI models can read captchas. You can also setup cloudflare to bypass WAF rules in case you own the website. :) However always respect the rules from robots.txt to stay ethical.

@ridwanray 5 ай бұрын

Good job, Brad. I used your JavaScript content back then

@soniablanche5672 5 ай бұрын

On linux, puppeteer might not work unless you download the browser manually. There's a script inside the puppeteer package to download it but I don't remember which was it was

@Usef88 Ай бұрын

Using Ubuntu and it's working fine for me !

@abourinadtech 2 ай бұрын

thaank you veeeeeeeeerrrrrrrrry muchhhhhhhhhhhhhhhhhhhhhhhhhh

@TKomoski 5 ай бұрын

In the old days I used YQL and XPATH to scrape web sites

@thomasleong3510 2 ай бұрын

does (Selenium + pyautogui) same with (Puppeteer & Beautiful Soup)?

@pedroalabart8179 4 ай бұрын

Why is your console log logging into the termianl and not on the console of the browser?

@serychristianrenaud 5 ай бұрын

Thanks ...

@MasumKhan-f2l 4 ай бұрын

Thanks

@sjn_ 3 ай бұрын

How would we go about rendering the fetched data using the elements on the client side? I wouldn't really want to send the HTML elements through the API response. Kinda reminds me of the OG php days lol. Is that even possible using HTMX?

@lordsnow4029 4 ай бұрын

I would like that you start MEVN series and create playlist 😊 (i saw there is both MERN and MEAN stack playlist but not MEVN) pls pls start that 😅

@srdjagunjic 5 ай бұрын

How is this an ultimate guide if there is no tutorial on how to deal with dynamic content, rate limiting and robots.txt?

@soniablanche5672 5 ай бұрын

imagine caring about robots.txt lmao

@anothermouth7077 5 ай бұрын

Then you become more creative on your own!

@kinotkuenterprise7372 5 ай бұрын

Share your tutorial then.. haha

@coolestCatEver 5 ай бұрын

You can scrape dynamic content with puppeteer

@BenChanan-o7g 4 ай бұрын

Can we get PHP?

@sonishn5222 5 ай бұрын

From crashcourses to guide. Wow. Thanks

@taxzanUSA 4 ай бұрын

I want to scrape multiple tables from a website search query into Excel. where do i begin?

@RickBeacham 2 сағат бұрын

I would use Google's Sheets. You can use JS with it. Or just use a CSV file then convert to EXCEL.

@irfanaliboina2805 4 ай бұрын

Nice

@greendragon0009 5 ай бұрын

Reverse racoon days. Am I right. :)

@swojnowski453 5 ай бұрын

My good advice, do not scrap other people's work. It is there for you and others to learn from, it is not there to feed LLMs.. People do not understand that, hence they get 403 status as soon as I notice a scrapping attempt.

@soniablanche5672 5 ай бұрын

that's why you use proxies and fake user agents :^)

@tooeasy852 5 ай бұрын

@@soniablanche5672 they will found you dummy.

@swojnowski453 5 ай бұрын

@@soniablanche5672 it will not work. Both of these can be blocked easily. If not logged-in-user access can be implemented or even paid access. By scrapping you push information people need behind extra safety measures and remove affordability for those who can't afford books. Do not do that, you are depraving people of access to info they might badly need. In short, you are killing the internet as we know it today.