Intro To Web Scraping With Puppeteer

In this video, we will look at Puppeteer to scrape data from a web page.
💻 Code:
github.com/bradtraversy/cours...
Puppeteer Docs:
pptr.dev/
⭐ All Courses:
traversymedia.com
💖 Show Support
Patreon: / traversymedia
PayPal: paypal.me/traversymedia
👇 Follow Traversy Media On Social Media:
Twitter: / traversymedia
Instagram: / traversymedia
Linkedin: / bradtraversy
Timestamps:
0:00 - Intro
0:36 - Install & Setup
3:36 - Init Browser & Page Object
5:02 - Screenshot & PDF
6:54 - Targeting HTML, Text, and Links
11:22 - Scraping Courses
17:08 - $$eval()
18:40 - Save JSON Data

[@vitorbutkus]

UPDATE! As for 2023 May, you'll have to change #courses for #cscourses, otherwise the code will return an empty array on 14:00.

[@straktormedia8676]

Thanks for sharing this small time-saving detail! :)

[@orlundoexplainsai]

Thank you for this!

[@lucareichelt7338]

still getting an empty constant back. Did he/his hoster somehow disable it due to drain on the website?

[@ruthstorm8905]

I watched and coded along 4 videos and finally, thanks to this one, I can understand this topic. Very clear and concise! I am working hard to become a front end developer and I have a test ( for a job) on web scraping next week. I feel ready now!

[@misterdanger4727]

Super awesome man, i searched the whole you-tube, but i found your explanation the best.

[@christopheanfry2425]

As usual another content so we can learn new things. Thanks Brad for your dedication you help us so much not only on how to code but with your open minded regarding all the aspect of programming and all the possibilities that contains. 🙏🙏

[@kleczekr]

This is a good intro. If you look into regular Puppeteer for scraping tutorials they often go into major projects which end up going out of date very quickly; I really enjoy your review of the basics, it makes it very accessible and easy to refer to.

[@lifelessalarm]

Thanks for the video! This is great stuff. I used Puppeteer for a project at work, scraping charts from a web app, taking screenshots, and saving data into a .csv file. Very handy tool.

[@qwizzwizz]

I wish everyone can make tutorials of this quality.

[@964tractorboy]

A great video. One of the best 'scraping videos I've seen on KZbin which starts from the ground up.

[@FitKannadiga]

This can change a lot in how the migration of sites happen esp on UI. Happy to learn this

[@karsongrady]

Thanks, Brad! An "advanced" puppeteer tutorial would be awesome. I've wanted to combine scraping with a database that updates on cron to create a custom, one-off API with just a couple files. Just an idea. Thanks again for all your content!

[@henrymunoz2035]

This was amazing thank you so much Brad hope all is well with you and the Beautiful family,

[@war4tek]

Thank you Brad! I appreciate you so much. Thank you for your dedication to helping others.

[@jamalahmad8048]

Such a quality content you're providing for free thanks brad sir ❤️

[@jeremyhorne6244]

Brad Schiff introduced me to Web Scraping. Great vid.

[@ricardocnn]

Awesome! Best tutorial about web scraping. We need more about this topic Brad!

[@jameshansen801]

This is so great to see! Not too long ago I got my 1st dev role that required the use of puppeteer the majority of the time. Knowing absolutely nothing about it, I was pretty much thrown in with the wolves. But it was such an awesome learning experience!

[@user-nn5vq5hl4q]

Helpline📲📥⬆️ Questions can come in⬆️

[@ovuokeaghwotu2389]

Thank you for the tutorial! In my case when creating the PDF, I included the 'fullPage' option to make it work.

[@gideonodiokineh3145]

Thank you @Brad for this awesome video.

[@kierramalone4968]

This is a great video. It's easy to follow along and understand.

[@thinotmandresy]

If you take a look at my search history, you'll find out that I was searching for scraping tutorials 2 days ago. I'm super happy that you released this video today. The timing is just perfect. Thank you so much!

[@amanuelgebremariyam9751]

Same here 😁😁

[@JeanDidier]

This happened to me a few months ago and I was just curious about it 🤔 Law of attraction in action?

[@tnktaliaba2994]

Damn me too lol Brad is the best !

[@toddbrannon6262]

Awesome video Brad. I wanted to comment because web scraping has great use cases in the real world. I am a BA with an agency that works with a very large client in the news/journalism space (one of the largest, in fact) and I'm working with a developer that usually creates new story feed ingests for their API so they can sell/syndicate the stories out to other news outlets all over the world. This is usually done via an XML feed but this one in particular is just links to HTML pages so the developer is scraping the stories from HTML and adding them to the API (this one for soccer/football stories related to the 2022 World Cup). So creating your own API by scraping data for a particular niche or use case is quite a valid skill set to have.

[@JeatBunkie]

Interesting

[@akashwhocodes]

Thank you so much. I was finding it really hard to find error in my code, and as soon as I saw the screenshot method in first 5mins of your video, I tried it and got to know the error and mistake I was making. :)

[@aadityaghag8085]

My man is back with the tutorial I wanted !

[@anishjoshi1999]

Congratulations on 2 million subscribers, Brad! The whole tech community is proud of you.

[@Julesarixx__my__teleqram__ID]

👆send a direct message for support and guidance .

[@edh9500]

Thank you, Brad. Super easy video to get me started with Puppetter.

[@Hacking-NASSA-with-HTML]

Thank you, Brad! You are Rock, as always 👍

[@ahmad-murery]

That was a really well made video as usual, Thanks Brad 💯👍

[@tombenbow100]

Great tutorial. Thank you for uploading this.

[@narendrajoshi5460]

Hi Brad! how are you? Great to watch your video after long time. You still inspire me.

[@paulthomas1052]

Great tutorial for a really useful library. Thanks. Also....for anyone getting time-outs due to slow connections etc.....add this line before the 'goto' await page.setDefaultNavigationTimeout(0);

[@trammellwebdev]

Wow puppeteer is awesome! Will definitely be playing with this soon ❤

[@CharlesBLim]

Whoa! That went smooth.... Thanks for the tutorial..

[@cannabisanomaly]

another fantastic vid, brad! you're a real one. i wanted to point out on the fs.writeFile() part, you can make it so that the JSON.stringify() method will automatically format the output in a readable way by including a third parameter; in this case, the length of array of objects. null is used to bypass the optional second 'replacer' parameter: JSON.stringify(courses, null, courses.length)

[@codified1]

Thanks.I am automating my work with beautiful soup.

[@ssk7690]

just realised you've gotten fit. Nice work man!

[@codewithfarhad8594]

great video as always. personally, I think there are loads of great scraping tools already invented (web scrapper io, octoparse etc) both free and paid that do pretty great job of scraping all kind of content and even allow one to create spiders & schedule scripts. everyone should know about those too! 😍

[@prgmaztoo]

Congratulations on 2 million subscribers.

[@Itailewinwin]

Right on time 🤘

[@nevengerencir6796]

Brad, I just bought two of your udemy courses - Node masterclass and react front backend 2022. I just came here to say big thanks man so far I am super satisfied. Thanks for everything you do !

[@tahamohamed2161]

I do not have the money to buy these courses from the Udemy platform. Is it explained here in the same way that it is explained on Udemy or not?

[@gavincoulson3900]

I love puppeteer, I made an actual product for a company that uses it and dang it’s so cool what it can do

[@JeanDidier]

Great value, appreciated

[@sabuein]

Thank you, Brad, very much.

[@collinsk8754]

Great tutorial! Very useful indeed. 😊😊

[@rangabharath4253]

Awesome as always ☺️

[@TheTeup]

Awesome tutorial Brad🤘 Your follower from Afghanistan😊

[@gbengaomoyeni4]

I have been trying to build a crawler for a long time but no success. With this, the possibility is endless. Thank you Brad. Your tutorial is always a top-notch. God bless

[@Julesarixx__my__teleqram__ID]

👆send a direct message for support and guidance .

[@yeminsonsanchez7173]

Gracias excelente video y muy bien explicado... ganaste un suscriptor de Latam...

[@djelasrca8091]

Nice video. Hope to see more about this topic. It's not easy to find good content about it

[@danielvega646]

Thanks a lot for your content!

[@nro337]

Awesome video as always!

[@cloudsystem3740]

thanks for the guide

[@javadmh7827]

Great Tutorial, Thanks

[@1godlessmonkey]

You're looking healthier, Brad. Hope you're working out and staying strong.

[@sfey93]

Nice explanation. Thanks :)

[@cloud89gaming97]

Awesome thanks 😊

[@abhis3kh]

Awesome. Thank you very much :)

[@novailoveyou]

Brad is the best!

[@krishnaSahu-dy6ho]

Really Great Video

[@akitikorisegun2653]

I WANT TO USE THIS OPPORTUNITY TO SAY A HUGE THANKS BRAD, YOU'RE THE BEST, YOUR TUTORIAL IS AMAZING AND EASY TO UNDERSTAND, YOU'VE HELPED A LOT OF PEOPLE WITH YOUR TUTORIAL, AND IM NUMBER ONE. PLEASE MAKE A VIDEO FOR RESET PASSWORD. I HOPE YOU READ MY COMMENT. THANKS BRAD

[@RaviGupta-ut9vy]

Awesome, I'm waiting for in depth course.

[@user-nn5vq5hl4q]

Helpline📲📥⬆️ Questions can come in⬆️

[@kumardeepam]

Awesome video!

[@aimattant]

Incredible - thank you. Completed it. I plan to go into this further until one-click integration. One question - how would you scrape through all of the website pages in an index with Puppeteer, from page 1 - page 100 - scraping the content? And then adding this to. a CVS file later - pandas? Would appreciate your help.

[@mnageh-bo1mm]

Great Tutorial : )

[@tadakuniyasuda8214]

You surprise me everyday.

[@Movies_15Tv]

Hey Brad first of all thank you for you videos,skills you give us and I would like to ask solid.js crash course,thank you.

[@bassamry]

good video, subbed!

[@walaazahran29]

wow amazing

[@riteshkumarSanampur]

Love u sir

[@asken5139]

Such good marketing of your courses:-) You should put a puppeteer course on your website, and then basically redo this video😅Might have a nice conversion🙂

[@thomasnarkiss6319]

No "promos". Yet awesome. Thanks, Brad. P.S. (Dec 2023) #courses > #cscourses P.P.S Advanced scrapping tutorial will be amazing.

[@nmugumya]

♥

[@syntaxed4365]

3 minutes and 240 views... early early. Good stuff as always, Brad!

[@Stars4Hearts]

I got a notification hmm 🤔

[@syntaxed4365]

@@Stars4Hearts Congrats on being subbed.

[@Stars4Hearts]

@@syntaxed4365 vote and bring friends 2024.

[@drucifer6]

I JUST used this for some critical css scraping. What are the chances!

[@bassamsami7319]

great work , can we download pdf files or videos from website using this nice tool? thank you

[@soniablanche5672]

never forget to close the puppeteer browser. I had a web server constantly crash because we didn't close the browser when an error occurred so we kept opening browsers without closing the ones we stopped using, the server's memory eventually saturated. Lesson learned : always close the browser in "finally" in the "try/catch/finally" blocks

[@user-nn5vq5hl4q]

Helpline📲📥⬆️ Questions can come in⬆️

[@hermesmercuriustrismegistu4841]

how can you scrape handlerbars injected values into HTML? thx for the help

[@JACKoPL]

Interesting package. Maybe I trying using this for laravel ;) If I can...

[@deanemarks8611]

Thanks for talking like a normal person. Refreshing

[@Julesarixx__my__teleqram__ID]

👆send a direct message for support and guidance .

[@mausaedof8398]

How would you scrape dynamically created classes?

[@asvicnal8483]

Would you be able to put together a crash course on DynamoDB and HTTP Module?

[@Doug87969]

Does anyone know how to easily have a variable copied to clip board from the .js and posted into a website?

[@Kischenwithayesha0]

what is the name of that vs code theme. I like that setup

[@Kodeispoetry]

Thanks, Can you make more courses on puppeteer scrapping in detail + there are ni convincing courses for developing chrome extensionsin market. In you can make in-depth course for Chrome extension. Thanks.

[@coleogden907]

Hey Brad, long time viewer and have taken a few of your udemy courses. Any plans for a deployment series explaining how to properly set up and deploy full stack apps across hosts like AWS, Azure etc...?

[@shauntonesify]

Yes, I want this too. :)

[@asimullahm.phil.scholardep2519]

Is there any tool or technique for automatic web scraping without directing towards any specific website. And Is this concept crawling or scraping in which automatic website elements are being returned without specifying towards any single website.

[@tyrojames9937]

👍🏾

[@akshatmishra8664]

After 11:00 whatever I'm trying to do I get the following error node:internal/process/promises:288 triggerUncaughtException(err, true /* fromPromise */); Can someone help

[@johnnwabuforudemezue1108]

How can we use this to build a bot

[@JimKernix]

Anyone know how he got the PDF format list to popup?

[@Mrbigdraco]

how can i do this but with websites that have "paste URL here" with my own URL and get a screenshot of the new page.

[@MohtasimHasan]

can i open puppeteer in my used browser

[@firelordkushroll]

i just cant seem to grab the right thing i need. it has complex class names and div structures i cant seem to crack it.

[@1000ylovers]

Can this also scrap meta data of the websites??!

[@thongnguyen5024]

Is that new "Traversy Media" animation? :D

[@trongnhanle83]

Can we scraping data from Facebook ads ?

[@1000ylovers]

I am just wondering that instead of doing puppeteer on nodejs, can I do this on nextjs project? I believe Next.js is sort of combination of React + Express; I can write backend-related logic.

[@user-nn5vq5hl4q]

Helpline📲📥⬆️ Questions can come in⬆️

[@goodluckoriuwa1669]

i have tried this too many times but i still get a timeout error. can someone help me fix this? 'TimeoutError: Timed out after 30000 ms while waiting for the WS endpoint URL to appear in stdout!' is the timeout message from the command prompt

[@johnm8358]

what about logging into websites?

[@developedbypy]

can you please continue this series?

[@Julesarixx__my__teleqram__ID]

👆send a direct message for support and guidance .

[@Festivejelly]

Why not playwright instead?