Intro To Web Scraping With Node.js & Cheerio

In this video we will take a look at the Node.js library, Cheerio which is a jQuery like tool for the server used in web scraping. This is similar to the python video that I did on web scraping
Sponsor:
DevMountain Bootcamp - goo.gl/6q0dEa
Code For This Project:
gist.github.co...
💖 Become a Patron: Show support & get perks!
/ traversymedia
Website & Udemy Courses
www.traversymed...
Follow Traversy Media:
/ traversymedia
/ traversymedia

[@IsaacAsante17]

Brad, you always post videos that we need. This tutorial was brilliant.

[@TraversyMedia]

For people asking what web scraping is used for, these may be helpful www.quora.com/What-are-examples-of-how-real-businesses-use-web-scraping-Are-there-any-types-of-businesses-which-use-this-more-than-others www.quora.com/Why-is-web-scraping-still-important-in-2017-What-are-some-popular-use-cases And please understand guys, when I do a video, it is just me showing you how to do something, it is up to you if you find it useful or not. I am here to help people learn not tell them what to do. Please understand that. I want to cover all areas of web development/coding, this happens to be one. People seem to think when I do a video I am saying you need to do this, no it is just a resource for people that are interested or just a fun little project

[@ridl27]

Nice, Brad! Very Useful tutorial for me! It took me ~1,5 hours to write the program my dad needed in his job and it will save a lot of time for him. Thank you!

[@savannahniemeyer]

Thank you, Brad. I can't ever follow my coding bootcamp instructor in class, so I come to your videos after and everything begins to make sense. Lifesaver!

[@DerJP]

This is so helpful. Gonna create a tool which scrapes a forum where I am active in. Every time a new post is created, I will make it send me an notification. The forum doesn't have this feature, so I will always be the first one to respond. :D

[@Toochilledtocare-_-]

DerJP great project idea

[@DerJP]

@@Toochilledtocare-_- thank you :)

[@Rohsn1]

Brad your videos must be some king of God's gift for me. For a personal project, today I was searching frantically for one on Scrapping with Cheerio and Node.. and here I have from you.. Thanks a million..

[@indiajackson5959]

Best video I'v seen thus far! You have no idea how essential this was to my phd. I will be acknowledging you in my dissertation.

[@itzyuzuruclips]

This tutorial pretty legit. I tried learning cheerio a lot for few months but couldn't really understand much no matter what. Then came across this tutorial like few days ago and it made it look so easy :D

[@alexz9499]

You spent a lot of time on one single node package and didn't understand much after months? Maybe look for something else rather than programming. Sorry, but it's true.

[@itzyuzuruclips]

​@@alexz9499 Thx for the advice, tho imma kindly decline it :) Tru, I'm not a full stack developer or of any sort but I've learned a lot in these last 2 years, stuff which I wouldn't have without programming. It's something I love to do and will continue on doing cuz it has become a part of my passion

[@porter2k7]

Wish for you every happiness Brad. You help people SOOO much dude and I just wanted to express my huge thanks & unending appreciation. Waves & Smiles from Scotland :)

[@albinoporchagezzee4902]

I was actually struggling for hours with page scraping. Thank you very much.

[@jeankule5925]

Thank you dude i've been looking for that thing for hours and i've finally got what i needed,you're a life saver !

[@Liquiddeath100]

I literally just sat down to do this using node and cheerio. Awesome timing. Thanks for the video.

[@michaelolukaka]

still learning a lot from you...you are ever my first choice of a tutorial when I search for help...thanks man

[@jagadeeshmanoharan9613]

Just now completed your react front to back course. And the next one is here. Boosting up💪

[@TraversyMedia]

Awesome!

[@jagadeeshmanoharan9613]

Keep going Brad. You are the inspiration for web developers like me.

[@mayankk2800]

How is the course??

[@jagadeeshmanoharan9613]

mayank k undoubtedly the best efforts from Brad .

[@KochharAmandeep]

Learning React Front to Back ❤️

[@bobdpa]

Yet another save by Brad. I would've had to pay someone to pull this off. Thank you!

[@codenx2]

You have been always to the point. My whole internet search ends here

[@WikkiOnIlluminati]

I was about to do some scraping. Normally I do in PHP. But this time i was thinking to do in Node and MongoDB. Brad thank you so much for this perfect tut.

[@jcs7757]

dude, you explained it very well, it was THE best explaining about web scraping i found on youtube. i could keep up and understand though im a starter programmer, but its not yet what i seek. i'm looking for an introduction tutorial on the basics of these methods and parameters. an explanation about what every instance of the code will do. i don't even know if that is possible.

[@munapadhi8723]

Traverse media is a one of best new learning

[@TheChowdhouryS]

Traves I love your YouTub and Udemy lessons, please continue creating detail videos for a new developer like me.

[@kamalhm-dev]

Thanks for all the help you provide, Brad, hope you live long and prosper!

[@neithanm]

A tiny suggestion: instead of reading '(i, el)' like "index and el", you should say "index and element". Everybody can read "el" but not everyone may know that it's an element and cause confusion.

[@Tidaltwist]

I think It's obvious that the .each function takes in the index and element arguments. People who don't know may quickly google it.

[@theblinkstykrab3106]

@@Tidaltwist I think it's obvious that the ".each" function is actually called ".forEach". People who don't know this would quickly realize after trying to google search for it

[@honglytech]

Who thinks el = else? :D

[@linyerin]

@@honglytech To me 'ele' is more obvious as 'element'

[@GeekMustHave]

Brad does vacation, I'd more like to believe in time travel. Happy Trails, Keep broadcasting!!

[@omiros210]

Excellent description of Web Scraping! Keep up the good job!

[@reidyoung298]

12:36 "Now notice there's a shit load of white space there..." Brad, you're awesome, man. Love your videos. Thanks for keeping it fresh and consistent. You help to keep my curiosity flowing, that's for sure. Any chance you can hop into some buzzword spaces like blockchain, AI, NLP, etc.? That would be sick to check out if it makes sense for your audience.

[@transforms-tools]

I was always curious about Web Scraping and you made really easy for me to understand. Thanks alot

[@YaserAz]

Thank you Brad. Have a nice vacation my friend, all greetings to your family .

[@MarkZuckerberg69691]

getting scary now, everytime I struggle with some javascript problem, you make a video about it within the same day. thanks, gonna scrape the shit out of those websites now

[@TraversyMedia]

Mark, I would think you would know how to scrape a site, with you know , building the worlds biggest application and all :)

[@MarkZuckerberg69691]

Traversy Media haha you humans dont know about time travel yet, funny. I aquired information from your channel and made myself rich 15 years ago

[@MuffinologyTrainer]

Real nice, looking forward to more 'ethical' courses like this one, I needed web scraping for so many workarounds. Even if its considered "bad" because you could easily scrape off and use data from public websites, it's still a tool available for the public and makes lots of tasks way easier, if you find the right use for it.

[@greddee]

Mad Trippin I agree and I think that webscraping is okay, as long as you do it like once a day/week or whatever and keeping it at a minimum.

[@AceixSmart]

@4:44 "const money sign"😂😂😂

[@ResonanceMusic2011]

Awesome tutorial, Brad. Super happy to do some js scraping!

[@chamodnimsara7054]

u da man Brad ....I just wanted this video from u and there it is thanks a lot man

[@Eshcole]

This was fun, simple, and awesome. Thanks Brad!

[@MULTICODE]

Thanks for this upload, you are helping a lot. And I appreciate it. 😍 traversy media is my web dev reference favorite

[@KingstonFortune]

very well explained, you're the best

[@williamwilson9866]

Good introduction to cheerio. On csv files, if you surround the date value with double quotes on the write statement, you could avoid replacing the comma.

[@sray118]

Thank you so much traversy media, this solves my question

[@erkb9398]

This was very usefull and exactly the level I've needed to get started.

[@gtisonumehta5371]

I love your all videos and also your voice 😊

[@feranst9111]

This is actually very cool and helpful!

[@macmacmac740]

quick and easy to understand. Thanks.

[@siddhantkumar7036]

Enjoy your vacations and wouod love to see your photo with your family

[@nagahumanbeingzooofparticl8836]

Web Scraping with node.js.. Yes! Thank Brad

[@harmit5445]

enjoy your vacation my code guru

[@guyma15]

Amazing tutorial Brad. I was planning on making a website that requires NFL data. This is extremely helpful

[@ajitsakri9888]

i think it is dynamic but you need to host it somewhere(i guess)

[@Huy-ed2re]

I love this tutorial! Thank you, Brad!

[@chrismorris5241]

Best tut I have found thanks!

[@andreiconstantinescu9830]

Hi! Not here to pick on stuff, just curious about why did you choose a library powered by jQuery? I personally have not invested time in learning jQuery and I believe now (2018) it is still used, but for legacy stuff, mainly. Thanks for all the work and dedication, support and all the great content you put out there for us Brad, Happy Holidays! Andrei

[@TheUKFishingGuy]

Thats a well interesting video... thanks for putting in the effort to make a video and sharing

[@ldhpro]

thank you Sir.. It is good to see you do your best. so good.

[@Mishka2Boom]

Thank you mate! Super useful!

[@Quniverse]

God bless you and your channel

[@paolodrago9254]

Love these vids, you're the best my guy!!!!

[@sukanyasinha3583]

Sir, your videos are awesome... please upload some more videos on cheerio

[@darshankhandelwal7290]

you saved my day man

[@juanpaul6141]

Lovely tut. Enjoy vacations.

[@MegasXLR]

Great tutorial, thanks :)

[@ojo-femioma-victor9912]

Traversy Media is the best

Congratulations for this awesome video!

[@jakep7980]

This was very helpful, I was stuck with my personal project but I was able to learn and adopt the concept into my own code thanks!

[@DigitalMonsters]

How do you scrape a site that just has crazy amounts of nesting, duplicate elements, slightly different page structures between items, little to no ID's or class names? Basically really messy poorly / variably structured data?

[@siteyekad6683]

that was awsome man thank's very very much

[@HK-sw3vi]

I took a sip of water everytime he said "it's simple jquery stuff". update: I'm at the hospital being treated for water intoxication.

[@sadafarafat4887]

thank you sir for teaching us 😇😇

[@ego_sum_liberi]

It’ll come in handy. thanks.

[@vaggelismanousakis6147]

What happens if it returns null no matter what # (id) or .(class) you add to line 8 ? Can a website block Cheerio ?

[@craigdavies6937]

Another great tutorial. 😎

[@uyufokokim]

Wow, thanks so much for your help!!!

[@ayushdixit3183]

nice specs. brad....lov ur videos

[@onecarwood]

Aim eating Cheerios right now. I think it’s helping!

[@TraversyMedia]

HAHA!

[@nikitakrulikov]

Very Great and very awesome. Your video is really helpful for me. Thank you. I hope you'll make good video in the future. Thanks again. Best Regards. Lyov Orlov.:)

[@henriqueuller6865]

It's amazing! Thanks a lot!

[@AbhishekKumar-mq1tt]

Thank you for this awesome video

[@pietrodellanotte]

All right Mr. Brad!

[@sheikhrashed1242]

I love you brad 😀

[@sveinsoermo9883]

BTW Brad...pls on offending, BUT U need more sun bro! ENJOY U'r vacation, I cannot enjoy anything without working! Thx a lot for fantastic stuff on U'r channel!

[@TraversyMedia]

haha, I have gotten plenty the past few days :)

[@ИльнарКамалетдинов-м7ю]

When you recording video, you speak simple, and i understand alost everything what u said :D

[@ranzorethor5295]

instead of using regex to replace the whitespace, could you also use jqueries .trim() as well to do the same thing? Or would that also remove the spacing between the words of the post as well?

[@luissebastianarribasplatac8195]

This is awesome! There's anyway to return a json file instead of a csv file?

[@mohammedserwan9844]

Hope You Happy Vacation😀

[@TraversyMedia]

MOHAMMED Serwan thanks 😊

[@mohammedserwan9844]

Traversy Media Am Glad That You Have Answered Me Twice😁❤️

[@martijnkorbee1635]

Thanks for your video! :)

[@josegomera9066]

Great tutorial, thanks

[@asdfasdfuhf]

More nodejs videos! Keep it up!

[@cfsolutionsforfree]

Bro will This work dynamically? I mean if I change source websites contents, will it automatically change the csv?

[@ajitsakri9888]

yes it is dynamic but we need to find a way to host it(i guess)

[@DuneKraftwerk]

Hi Brad, I know this is not related to this video and it is not the first time this question has been asked but, may I know how you made your KZbin logo and intro ? Everything looks so professional. cheers.

[@simplyfabian]

Really great video on this for beginners, I've also did an introduction on Web Scraping with Puppeteer for handling cases where the pages that you want to scrape need to be dynamically rendered 💻

[@yschin5663]

Thanks Brad, this is what exactly I looking for!

[@salargentieri4540]

Brother Brad! Dope video! I was hoping you could throw me the npmjs info on request cause I can seem to isolate which one you are using. Also how do you feel about axios vs request?

[@NotLudovico]

Hi Brad, wonderful video. Have you got any book of web graphics to raccomande?

[@harrisfpv]

very good video! what would you do if the page you want to scrape renders using react or any other javascript framework? because it only shows you the html part before it gets rendered

[@atikfahad]

you are crazy man! ♥

[@elliottharris4526]

Setting up the environment e.g. installing node.js and having VS studio properly configured to follow your steps is admittedly probably beyond the scope of this tutorial, as well as you admitting you are in a rush but it would have been really helpful for a beginner like me. Any chance you would do a follow-up (or have already done) covering this?

[@ufotofu9]

Yes , I am almost postie that Travers has videos look like that. You shooed search for tutorials on node js, JavaScript, npm and dependencies.

[@elliottharris4526]

@@ufotofu9 Thanks. I will have a look.

[@bioJockey1]

Isnt casting every element with $(el) Inefficient? I guess for a program this small it doesnt matter but if i want to scrape lots of pages (as in close to a million) do you think it wouldnt matter still?

[@MaxOnMaxxer90]

Is it possible to just use this code on the client side, when a user clicks on a button? Because it says that Cheerio is specifically for the server.. Thanks for the great video, thumbs up!

[@akram42]

pefect tutorial , cheerio is better than beautifulSoup when scrapping, I'm trying to do pagination with cheerio. thanks

[@abhijeetraj7952]

great lecture! by the way you can just use .trim() to extra spaces.

[@SiviweXakaza]

Can you create an express server that returns a json object of the scrapped data?

[@jonathanclark6489]

you are such a G!!!!!!

[@dcab7290]

Hi, I tried your code but neither it outputs the data on console nor in the file. The line after the "each" statement that is console.log("scraping done..") executes. Please help

[@djBulba]

Layman question, for sites that need login (or maintain sessions) can we run this kind of code?

[@Mr.Sanguine]

Yu can combine selenium with this to do what you need

[@StaySickGdl]

I've been looking into this also without any luck, my company websites ask for a certificate PIN I've tried with python and selenium but cannot make it work to send the keys, for people like me who does not have access to AD API or to an application API is almost impossible to make things work because of all the security and password requirements you get