A Guide to Web Scraping with Node.js

Learn how to build a web scraper ⛏️ with NodeJS using two distinct strategies, including (1) a metatag link preview generator and (2) a fully-interactive bot for Instagram. fireship.io/lessons/web-scrap...
1. Build a simple link preview scraper with Cheerio.
github.com/cheeriojs/cheerio
2. Build an interactive Instagram scraper with puppeteer.
github.com/GoogleChrome/puppe...
#javascript #nodejs #webdev
Take the quizzes 🤓
iOS itunes.apple.com/us/app/fires...
Android play.google.com/store/apps/de...
Upgrade to Fireship PRO at fireship.io/pro
Use code lORhwXd2 for 25% off your first payment.

[@oleksandrisaryk4728]

Me yesterday: Hmm...there is no API available, well, need to learn how to scrap with node. Fireship today: A Guide to Web Scraping with NodeJS Well done.

[@chicken_teriyaki]

No one like this comment, it has achieved greatness

[@BearologyLabs]

@Killian Malik Tried plenty of methods, this one seems to be working!

[@lizzagarcia6405]

L

[@PawelPolskiPL]

I love it! It’s the most useful video on KZbin for me, ever! Today all day I was making app using puppeteer, I didn’t know it’s so easy, thank you as always for great video. Now I’ll be saving over 40 min every day at work 😂👍

[@actualmortgage7122]

Thanks for this upload. I got into a little bit of web scraping a couple years ago but ran into a lot more complexity than I was expecting for a tiny little side project. Now that I know a lot more, and JS, hoping to add this to the toolbelt.

[@PaleSaturn]

This is one of the best programming channels on KZbin. Love the production on your videos!

[@remoteworkboard]

You must be a mind reader. I was struggling with my link preview generator all week! Thank you soooo much.

[@MrDots99]

I've been doing this for that past 2 days at work and you upload a video today haha , fortunately I got it working but its always nice to see you upload !

[@alexanderf7008]

I only start understanding things better after watching your videos. It's such a great explanation and never boring.

[@arisweedler4703]

I love it! :D such well-made videos. I appreciate how you explain the problems that packages solve instead of just trying to peddling hype

[@theblackharted]

Your content is top notch and so well done!! Easily my favorite programming channel

[@Tizmo-tt9ry]

when you say as hackers i feel so cool

[@Fireship]

Haha, web scraping always feels very hacky.

[@anonwithamnesia]

@@Fireship Which font do you use?

[@Fireship]

@@anonwithamnesia Fira code

[@StingSting844]

Haha yeah its hacky and cool. Once you start to make money off scrapped pages you will sued though

[@vaishnav4750]

😊yes

[@armaandhanji2112]

HUGE fan of your Node videos. Thank you so much! Best channel on KZbin.

[@firaskudsy]

Thanks for those amazing videos... u r covering the full full stack 👍

[@almostanengineer]

I love the fact your using Insomnia for your rest requests, when I started working with API, I was told to try Postman, and found it overly complicated everything, then I found Insomnia, so much cleaner and easier to use.

[@anuraghazra4772]

I agree

[@curiousnrd]

Fantastic video! I’m excited to test this out. Thank you for sharing this information.🙏🏾

[@f3lixadam]

Hi Jeff, thank you so much for your videos! We are building a startup and without your tutorials and explanations, I don't know if we could have ever gotten such a great insight how the mechanims work, especially for Ionic/Firebase. Much Love from Germany

[@Fireship]

Wow, that is awesome! Keep me posted on your progress :)

[@pepperpeterpiperpickled9805]

@@Fireship If I didnt think you were an AI, I'd want to buy you a beer

[@ninanordbo]

Love your content! Well done, my favorite youtube resource. Love that you get straight to the point.

[@bartub5369]

Great content! I have previously made a scraper for a discord bot with cheerio before, but never really fiddled around with puppeteer or anything similar to it earlier. Thanks for the ideas :)

[@TheGejr]

Just as I'm building a news-scraper you make this amazing video! You're using the exact same packages I am :^) Great video as always!

[@diabolo1]

OMG I’ve been looking for all the web for resources on WebScraping and suddenly your video appears! Well done!

[@boriskrstic]

This video is literally like you listened to me today what I was wondering about. Thanks for the puppeteer part! :)

[@Fireship]

Awesome! The topic came from a suggestion in the slack channel last week :)

[@AaronJack]

So many things could be built around puppeteer, and the await syntax makes it so easy to reason about. Also seems like this is a perfect use-case for cloud functions vs having a dedicated API server. Great video.

[@steaklover575]

Been here for a long time, your content just gets better and better!

[@ambarmutha8504]

🔥ship is AWESOME! Every time I have a project on my mind I find something useful to steal from this channel.

[@justafreak15able]

In an open source world there is no such thing as stealing.

[@tanujvyas6124]

Thank you for this video .I was looking for guidance for web scraping .This video prooved to be a great source.

[@jimmymac601]

As a long time subscriber, you need to make tutorials every day. Fantastic work.

[@Fireship]

I wish I could, my idea list is massive.

[@calebkopp4174]

Very insightful. Appreciate all the work that's put into the videos

[@Autoscraping]

An outstanding video that has been a valuable reference for our newcomers. We sincerely thank you for sharing!

[@camerenisonfire]

Always such interesting and well produce videos. Thanks, Fireship.

[@sebastiancuk7004]

Im using puppeteer in my work to make short videos testing the app automatically (feels awesome). Also really cool video keep the good work !

[@ShivamGupta-nz7dy]

When it comes to Web Scraping, most will talk about Python. Felt very happy knowing Web Scraping can also be done easily using JavaScript. Thanks a lot.

[@softwarelivre2389]

Javascript is the native environment for web scraping. Not only it is asynchronous, but also Javascript was created to be used on the web. To see people using interpreted, slower python for that is just sad. Not that there is something wrong with python, but it has other purposes. JS is much better for scraping pages generated with React, for example. On python, you can, at best, use a bot to simulate a user control. With JS, you can evaluate the DOM and then retrieve your data without the need of emulating a real user, and that is very powerful.

[@lyricalstudio7845]

All your videos are amazing!! Keep up the good work and we will do our best to support your channel. Cheers!! :-)

[@stevewitman]

Interesting topic and it was also helpful seeing a firebase cloud function being set up.

[@yejielwahnich6136]

Link preview I wonder if I will ever need this. Said jokingly knowing they talked about it just 1 week ago. Awesome content !!

[@Fireship]

I get my best ideas from Slack :)

[@yejielwahnich6136]

@@Fireship not for nothing we love/hang around slack ;)

[@randith87]

Just getting into Node, and Json. Having a lot of fun! Definitely a challenge! 😎💻

[@SmartWizzard]

Man you are awesome, all your videos are latest, useful and very informative. I very much happy that I have subscribed to your channel.

[@iykazorji8171]

This is some quality content yo! Thanks for this!

[@greatsuccess4734]

I have been waiting for this video for so long thank you so much

[@mateiadriel353]

the puppeteer support for firefox is awesome!

[@albertchung7641]

This channel's videos are all so cool. You rock!

[@alinandrei9006]

Thanks, understood what I came for from the first two minutes.

[@alokrawatt]

Thanks for your video, I have all of your videos. Loving it.

[@trusterzero6399]

As a programmer there is a lot of stuff that I kinda want to look in to but I never really get the chance to. These videos are helping me hard with that

[@igotapochahontas]

This is a really good video. Very clear explanation and simple

[@bharathravi7820]

Was looking forward to this, thanks a ton!

[@raph6709]

Dude...this is exactly what I needed... why are you so helpful

[@Oxcorp]

Love your videos man, best quality!

[@DarkNenyk]

Thanks to you I got into Angular

[@adante407]

This video is great! Just subscribed. Fantastic channel, amazing content. 👍

[@coolaydalena]

Ooohhhhwwww.. More Ideas about cloud functions please... 🔥❤️

[@htky]

Perfect timing, I just started a project that will involve scraping 👌🏻

[@drowmik]

That a cool detailed video in a very short time about scrapping...

[@will_abule]

Thanks Jeff this can be extended for ssr and seo 😁

[@iova666]

wish i had this vid a few years ago. you just gained a new sub :)

[@G3Number]

Your tutorials are top quality !!

[@zarefgamz2515]

You saved me a lot of time to find the name of a tool It has been more than 3 days searching for that tool

[@akarshbarar1492]

Hey one of the thing I have been searching from last 4 months..

[@iddrissraaj6737]

Great Video, informative as always

[@viveksoundrapandi]

Never new about puppeteer. Excellent one

[@RoshanKumar-yr1jf]

Thanks for the great video... I always wanted to do something in this domain...I was just shifting to python scrapy tool... Thanks again for awesome video..😃😃

[@jeandrepentz5011]

That puppeteer library looks neat, never knew about it

[@anonwithamnesia]

Great Video! I love your videos. Keep it up bro!!

[@aravind.a]

Good explanation with example, Jeff.

[@yallayeho1238]

Very interesting topic! Thank you!

[@alexgogan1617]

I'll be having such a scrape time with this!

[@milesmeow]

A great video for adding a tool to our web Swiss Army knife.

[@OniasdaRocha]

you just got yourself a subscriber, great content my man

[@MagnusVestergaard]

Web scraping is a great thing to know, thanks!

[@jlai383]

I like this kinda demo so much!

[@AngrejKumar]

this is awesome man. Thank a lot!

[@gavinmurphytheperson5446]

Thats pretty cool. Don't have a use case yet to use this yet but I can imagine how useful it could be. Great video as usual!

[@tworizki]

It is. Imagine you could scrape the entire net and let people browse your scraped pages using keywords. Oh wait...

[@tekforge]

Thanks for the insights!

[@aravindmj3723]

Thanks for the video Jeff..

[@gauravdasgupta3663]

Everytime I share some url in my LinkedIn posts..it loads the preview so beautifully...and I was always thinking how do they do that... U made it so easy.. now I can do it too... Thanks a ton

[@vinukurian389]

Exactly what i was looking for 👨‍💻

[@zdravko6t5]

Very interesting video. Learned so much from this video. Didn't know you can do such idea using Instagram.

[@T3P]

I really love watching your videos ❤️

[@GavinLon]

Very interesting video and very well presented. Thank you.

[@juancamiloq1]

Awesome content bro. Congrats!

[@RoyRope]

Awesome just scraping for a project right now.

[@firestoreprojects6202]

Amazing as always.

[@troykoss]

Great stuff, Jeff!

[@alemaoBOSS]

Love your content, just waiting for a few more courses to be available in fireship to buy pro. I will win the next T-shirt btw :)

[@mksoftwaresolutions9303]

Another great video. Thank you.

[@SouthPeter98]

Great video, as always!

[@sachinprabhuk6241]

Awesome videos. Keep up the awesome work

[@lifasibiya4810]

Thanks again for the video Jeff. Last year, I was using a library called HtmlAgilityPack to achieve some of this with C#. You've made it quite simpler. YOU DESERVE THE FIRST FLYING CAR MAN👍🏾

[@williamragstad]

Amazing video! Love your content!

[@ryansamarakoon8268]

This is great! I want to try make like an app that uses face recognition to try find pictures of you where you were tagged or not, but I was too late to start using the API. This now gives me hope that it's possible

[@fakesingh3235]

I was looking for this only!!

[@rockwoodfrank1517]

Very useful. Thank you!

[@xit]

That Tshirt Looks SICK!

[@scottmccartney1518]

Awesome vid, really appreciate it!

[@bjerz23]

Your videos are just so good

[@firstprincipleslearning]

Great videos as always !

[@DisasterSPA]

Amazing video as usual!

[@mohammedgadi884]

Awesome Explanation.

[@francis584]

awesome! thank you for that content.

[@adrienlucbert2769]

Clean and interesting video, as usual ;)

[@antogyn6124]

thanks for this!

[@BrianClincy]

Kevin Darrent.. I have used puppeteer and did a meetup with it. Thanks!