Intro To Web Scraping With Puppeteer

In this video, we will look at Puppeteer to scrape data from a web page.
💻 Code:
github.com/bradtraversy/cours...
Puppeteer Docs:
pptr.dev/
⭐ All Courses:
traversymedia.com
💖 Show Support
Patreon: / traversymedia
PayPal: paypal.me/traversymedia
👇 Follow Traversy Media On Social Media:
Twitter: / traversymedia
Instagram: / traversymedia
Linkedin: / bradtraversy
Timestamps:
0:00 - Intro
0:36 - Install & Setup
3:36 - Init Browser & Page Object
5:02 - Screenshot & PDF
6:54 - Targeting HTML, Text, and Links
11:22 - Scraping Courses
17:08 - $$eval()
18:40 - Save JSON Data

[@vitorbutkus]

UPDATE! As for 2023 May, you'll have to change #courses for #cscourses, otherwise the code will return an empty array on 14:00.

[@straktormedia8676]

Thanks for sharing this small time-saving detail! :)

[@orlundoexplainsai]

Thank you for this!

[@lucareichelt7338]

still getting an empty constant back. Did he/his hoster somehow disable it due to drain on the website?

[@misterdanger4727]

Super awesome man, i searched the whole you-tube, but i found your explanation the best.

[@qwizzwizz]

I wish everyone can make tutorials of this quality.

[@anishjoshi1999]

Congratulations on 2 million subscribers, Brad! The whole tech community is proud of you.

[@Julesarixx__my__teleqram__ID]

👆send a direct message for support and guidance .

[@christopheanfry2425]

As usual another content so we can learn new things. Thanks Brad for your dedication you help us so much not only on how to code but with your open minded regarding all the aspect of programming and all the possibilities that contains. 🙏🙏

[@thinotmandresy]

If you take a look at my search history, you'll find out that I was searching for scraping tutorials 2 days ago. I'm super happy that you released this video today. The timing is just perfect. Thank you so much!

[@amanuelgebremariyam9751]

Same here 😁😁

[@JeanDidier]

This happened to me a few months ago and I was just curious about it 🤔 Law of attraction in action?

[@tnktaliaba2994]

Damn me too lol Brad is the best !

[@ruthstorm8905]

I watched and coded along 4 videos and finally, thanks to this one, I can understand this topic. Very clear and concise! I am working hard to become a front end developer and I have a test ( for a job) on web scraping next week. I feel ready now!

[@lifelessalarm]

Thanks for the video! This is great stuff. I used Puppeteer for a project at work, scraping charts from a web app, taking screenshots, and saving data into a .csv file. Very handy tool.

[@kleczekr]

This is a good intro. If you look into regular Puppeteer for scraping tutorials they often go into major projects which end up going out of date very quickly; I really enjoy your review of the basics, it makes it very accessible and easy to refer to.

[@toddbrannon6262]

Awesome video Brad. I wanted to comment because web scraping has great use cases in the real world. I am a BA with an agency that works with a very large client in the news/journalism space (one of the largest, in fact) and I'm working with a developer that usually creates new story feed ingests for their API so they can sell/syndicate the stories out to other news outlets all over the world. This is usually done via an XML feed but this one in particular is just links to HTML pages so the developer is scraping the stories from HTML and adding them to the API (this one for soccer/football stories related to the 2022 World Cup). So creating your own API by scraping data for a particular niche or use case is quite a valid skill set to have.

[@JeatBunkie]

Interesting

[@964tractorboy]

A great video. One of the best 'scraping videos I've seen on KZbin which starts from the ground up.

[@jameshansen801]

This is so great to see! Not too long ago I got my 1st dev role that required the use of puppeteer the majority of the time. Knowing absolutely nothing about it, I was pretty much thrown in with the wolves. But it was such an awesome learning experience!

[@user-nn5vq5hl4q]

Helpline📲📥⬆️ Questions can come in⬆️

[@jamalahmad8048]

Such a quality content you're providing for free thanks brad sir ❤️

[@war4tek]

Thank you Brad! I appreciate you so much. Thank you for your dedication to helping others.

[@ricardocnn]

Awesome! Best tutorial about web scraping. We need more about this topic Brad!

[@gideonodiokineh3145]

Thank you @Brad for this awesome video.

[@henrymunoz2035]

This was amazing thank you so much Brad hope all is well with you and the Beautiful family,

[@karsongrady]

Thanks, Brad! An "advanced" puppeteer tutorial would be awesome. I've wanted to combine scraping with a database that updates on cron to create a custom, one-off API with just a couple files. Just an idea. Thanks again for all your content!

[@kierramalone4968]

This is a great video. It's easy to follow along and understand.

[@aadityaghag8085]

My man is back with the tutorial I wanted !

[@edh9500]

Thank you, Brad. Super easy video to get me started with Puppetter.

[@FitKannadiga]

This can change a lot in how the migration of sites happen esp on UI. Happy to learn this

[@Hacking-NASSA-with-HTML]

Thank you, Brad! You are Rock, as always 👍

[@ahmad-murery]

That was a really well made video as usual, Thanks Brad 💯👍

[@jeremyhorne6244]

Brad Schiff introduced me to Web Scraping. Great vid.

[@cannabisanomaly]

another fantastic vid, brad! you're a real one. i wanted to point out on the fs.writeFile() part, you can make it so that the JSON.stringify() method will automatically format the output in a readable way by including a third parameter; in this case, the length of array of objects. null is used to bypass the optional second 'replacer' parameter: JSON.stringify(courses, null, courses.length)

[@akashwhocodes]

Thank you so much. I was finding it really hard to find error in my code, and as soon as I saw the screenshot method in first 5mins of your video, I tried it and got to know the error and mistake I was making. :)

[@ovuokeaghwotu2389]

Thank you for the tutorial! In my case when creating the PDF, I included the 'fullPage' option to make it work.

[@tombenbow100]

Great tutorial. Thank you for uploading this.

[@nevengerencir6796]

Brad, I just bought two of your udemy courses - Node masterclass and react front backend 2022. I just came here to say big thanks man so far I am super satisfied. Thanks for everything you do !

[@tahamohamed2161]

I do not have the money to buy these courses from the Udemy platform. Is it explained here in the same way that it is explained on Udemy or not?

[@CharlesBLim]

Whoa! That went smooth.... Thanks for the tutorial..

[@trammellwebdev]

Wow puppeteer is awesome! Will definitely be playing with this soon ❤

[@narendrajoshi5460]

Hi Brad! how are you? Great to watch your video after long time. You still inspire me.

[@codified1]

Thanks.I am automating my work with beautiful soup.

[@paulthomas1052]

Great tutorial for a really useful library. Thanks. Also....for anyone getting time-outs due to slow connections etc.....add this line before the 'goto' await page.setDefaultNavigationTimeout(0);

[@ssk7690]

just realised you've gotten fit. Nice work man!

[@JeanDidier]

Great value, appreciated

[@prgmaztoo]

Congratulations on 2 million subscribers.

[@sabuein]

Thank you, Brad, very much.

[@Itailewinwin]

Right on time 🤘

[@collinsk8754]

Great tutorial! Very useful indeed. 😊😊

[@TheTeup]

Awesome tutorial Brad🤘 Your follower from Afghanistan😊

[@danielvega646]

Thanks a lot for your content!

[@codewithfarhad8594]

great video as always. personally, I think there are loads of great scraping tools already invented (web scrapper io, octoparse etc) both free and paid that do pretty great job of scraping all kind of content and even allow one to create spiders & schedule scripts. everyone should know about those too! 😍

[@yeminsonsanchez7173]

Gracias excelente video y muy bien explicado... ganaste un suscriptor de Latam...

[@rangabharath4253]

Awesome as always ☺️

[@nro337]

Awesome video as always!

[@djelasrca8091]

Nice video. Hope to see more about this topic. It's not easy to find good content about it

[@javadmh7827]

Great Tutorial, Thanks

[@cloudsystem3740]

thanks for the guide

[@gbengaomoyeni4]

I have been trying to build a crawler for a long time but no success. With this, the possibility is endless. Thank you Brad. Your tutorial is always a top-notch. God bless

[@Julesarixx__my__teleqram__ID]

👆send a direct message for support and guidance .

[@sfey93]

Nice explanation. Thanks :)

[@gavincoulson3900]

I love puppeteer, I made an actual product for a company that uses it and dang it’s so cool what it can do

[@cloud89gaming97]

Awesome thanks 😊

[@abhis3kh]

Awesome. Thank you very much :)

[@1godlessmonkey]

You're looking healthier, Brad. Hope you're working out and staying strong.

[@krishnaSahu-dy6ho]

Really Great Video

[@mnageh-bo1mm]

Great Tutorial : )

[@aimattant]

Incredible - thank you. Completed it. I plan to go into this further until one-click integration. One question - how would you scrape through all of the website pages in an index with Puppeteer, from page 1 - page 100 - scraping the content? And then adding this to. a CVS file later - pandas? Would appreciate your help.

[@tadakuniyasuda8214]

You surprise me everyday.

[@RaviGupta-ut9vy]

Awesome, I'm waiting for in depth course.

[@user-nn5vq5hl4q]

Helpline📲📥⬆️ Questions can come in⬆️

[@novailoveyou]

Brad is the best!

[@kumardeepam]

Awesome video!

[@bassamry]

good video, subbed!

[@thomasnarkiss6319]

No "promos". Yet awesome. Thanks, Brad. P.S. (Dec 2023) #courses > #cscourses P.P.S Advanced scrapping tutorial will be amazing.

[@samcutter7042]

3:44 - saving my place.

[@walaazahran29]

wow amazing

[@riteshkumarSanampur]

Love u sir

[@akitikorisegun2653]

I WANT TO USE THIS OPPORTUNITY TO SAY A HUGE THANKS BRAD, YOU'RE THE BEST, YOUR TUTORIAL IS AMAZING AND EASY TO UNDERSTAND, YOU'VE HELPED A LOT OF PEOPLE WITH YOUR TUTORIAL, AND IM NUMBER ONE. PLEASE MAKE A VIDEO FOR RESET PASSWORD. I HOPE YOU READ MY COMMENT. THANKS BRAD

[@Movies_15Tv]

Hey Brad first of all thank you for you videos,skills you give us and I would like to ask solid.js crash course,thank you.

[@syntaxed4365]

3 minutes and 240 views... early early. Good stuff as always, Brad!

[@Stars4Hearts]

I got a notification hmm 🤔

[@syntaxed4365]

@@Stars4Hearts Congrats on being subbed.

[@Stars4Hearts]

@@syntaxed4365 vote and bring friends 2024.

[@deanemarks8611]

Thanks for talking like a normal person. Refreshing

[@Julesarixx__my__teleqram__ID]

👆send a direct message for support and guidance .

[@bassamsami7319]

great work , can we download pdf files or videos from website using this nice tool? thank you

[@coleogden907]

Hey Brad, long time viewer and have taken a few of your udemy courses. Any plans for a deployment series explaining how to properly set up and deploy full stack apps across hosts like AWS, Azure etc...?

[@shauntonesify]

Yes, I want this too. :)

[@drucifer6]

I JUST used this for some critical css scraping. What are the chances!

[@asken5139]

Such good marketing of your courses:-) You should put a puppeteer course on your website, and then basically redo this video😅Might have a nice conversion🙂

[@hermesmercuriustrismegistu4841]

how can you scrape handlerbars injected values into HTML? thx for the help

[@asvicnal8483]

Would you be able to put together a crash course on DynamoDB and HTTP Module?

[@soniablanche5672]

never forget to close the puppeteer browser. I had a web server constantly crash because we didn't close the browser when an error occurred so we kept opening browsers without closing the ones we stopped using, the server's memory eventually saturated. Lesson learned : always close the browser in "finally" in the "try/catch/finally" blocks

[@user-nn5vq5hl4q]

Helpline📲📥⬆️ Questions can come in⬆️

[@JACKoPL]

Interesting package. Maybe I trying using this for laravel ;) If I can...

[@mausaedof8398]

How would you scrape dynamically created classes?

[@Kischenwithayesha0]

what is the name of that vs code theme. I like that setup

[@Doug87969]

Does anyone know how to easily have a variable copied to clip board from the .js and posted into a website?

[@nmugumya]

♥

[@asimullahm.phil.scholardep2519]

Is there any tool or technique for automatic web scraping without directing towards any specific website. And Is this concept crawling or scraping in which automatic website elements are being returned without specifying towards any single website.

[@Kodeispoetry]

Thanks, Can you make more courses on puppeteer scrapping in detail + there are ni convincing courses for developing chrome extensionsin market. In you can make in-depth course for Chrome extension. Thanks.

[@thongnguyen5024]

Is that new "Traversy Media" animation? :D

[@1000ylovers]

I am just wondering that instead of doing puppeteer on nodejs, can I do this on nextjs project? I believe Next.js is sort of combination of React + Express; I can write backend-related logic.

[@user-nn5vq5hl4q]

Helpline📲📥⬆️ Questions can come in⬆️

[@tyrojames9937]

👍🏾

[@akshatmishra8664]

After 11:00 whatever I'm trying to do I get the following error node:internal/process/promises:288 triggerUncaughtException(err, true /* fromPromise */); Can someone help

[@Mrbigdraco]

how can i do this but with websites that have "paste URL here" with my own URL and get a screenshot of the new page.

[@JensonMJohn]

Good Video. Btw, Web Scraping is Legal??

[@soniablanche5672]

yes as long as your not ddosing them with lots of requests per seconds

[@soniablanche5672]

It's also a good idea to check if these websites use public APIs to fetch the website data. If they do, it makes more sense to access that API than to scrape the website.

[@JensonMJohn]

@@soniablanche5672 I was pointing about the Situations where APIs are not available.

[@JimKernix]

Anyone know how he got the PDF format list to popup?

[@firelordkushroll]

i just cant seem to grab the right thing i need. it has complex class names and div structures i cant seem to crack it.

[@developedbypy]

can you please continue this series?

[@Julesarixx__my__teleqram__ID]

👆send a direct message for support and guidance .

[@goodluckoriuwa1669]

i have tried this too many times but i still get a timeout error. can someone help me fix this? 'TimeoutError: Timed out after 30000 ms while waiting for the WS endpoint URL to appear in stdout!' is the timeout message from the command prompt

[@1000ylovers]

Can this also scrap meta data of the websites??!

[@trongnhanle83]

Can we scraping data from Facebook ads ?

[@johnnwabuforudemezue1108]

How can we use this to build a bot

[@MohtasimHasan]

can i open puppeteer in my used browser