Build a Web Scraper (super simple!)

⭐ Sign up for my Full Stack Developer Course: www.codewithan...
In this video I show you how to build a Web Scrapper in a super simple beginner friendly way using Node.js.
Web scraping refers to the extraction of data from a website quickly and accurately. Many people move onto selling their web scraping tools for money, either by building them as a chrome extension or API, or selling them to data capturing companies. So, the option to make money off this tool, is there for you too.
Common questions:
- This tutorial assumes you have nodemon installed globally on your computer. If you do not, use command: npm i nodemon -g
Part 2 For Express Routing: • Get Data from Backend ...
Final code for Part 1 and Part 2: github.com/kub...
___
⭐ Use promo code ANIAKUBOW for 3 months free of WebStorm IDE here (I get no commission from this link, but am in a partnership): jb.gg/get_webs...
⭐ New to code and none of this is making sense? Watch my '12hr+ KZbin Coding Bootcamp' in which you will learn HTML, CSS and JavaScript Fundamentals completely from scratch. It's on my channel and its 100% free.
⭐ In most videos I use Tabnine as my A.I autocompletion tool. You can download it for free here (I get no commission from this link, but am in a partnership): bit.ly/tabnine-...
⭐ You can get a blockchain domain with my affiliate link here: bit.ly/get-a-cr...
⭐ If you would like to buy me a coffee, well thank you very much that is mega kind! : www.buymeacoff...
⭐ Sign up for weekly coding tips from my newsletter partnership: bit.ly/JS-tips
You can also find me on:
Twitter: / ania_kubow
Instagram: / aniakubow
#codingbootcamp​ #coding

[@josenardulli]

thanks Ania its working perfectly, may i ask why we cant use an arrow function inside the each on line 15 of the code when you call the cheerio ($) function i try it and i got all undefine but i cannot wrap my head around the why....

[@aniakubow]

I *think* it’s because cheerio is not configured to use arrow functions - but I can’t be sure - I haven’t looked into it enough :) I will have to when I’m back from holiday. I will pin this so others can see your great question 😄

[@aliabdelrhim]

@@austinps4026 This work for me: try { $('.fc-item__title').each(function () { const title = $(this).text const link = $(this).find('a').attr('href') articles.push({ title, link }) console.log(articles) }) }catch(err){ console.log(err) }

[@papajohnsuk5965]

because the arrow function scopes the function outside of 'this'

[@marcod.643]

You ca use an arrow function with the each method, but you must pass 2 parameters, an index and the element: $(".wathever", html).each((i, el) => { const title = $(el).attr("title") const image = $(el).find("img").attr("src") article.push({ titolo, image })

[@toddfisher8248]

Arrow functions use this from the calling context. Before arrow functions to do this you had to use the bind method when calling your function to switch the this context of your function. You could implement bind before it existed with apply when calling to have the function run with a different this… arrow functions just make what you usually want easier which is to carry the this context forward.. however many libraries exist that take advantage of the fact that the function you pass into another function can be called in away that changes the this…

[@revillsimon]

For anyone wondering… you’ll need to make sure the nodemon package is installed. You can run ‘npm i nodemon -g’ to use it globally on your machine. Or alternatively, you can run ‘npm i nodemon -D’ inside the project directory to use it as a development dependency while the project is running. Great video Ania, keep it up 👍

[@aniakubow]

Thanks for sharing this!!!! You are totally right I missed explaining this part- I will make sure to cover it in my next videos :)

[@alucard1532]

As a total newbie it saved me thanks.

[@SandraWantsCoke]

thx, but I like my demons

[@frogboy831]

thank u!! love when u have an issue that's way over your head and it's solved in the first comment u read

[@flexaeterna]

Ohhh that will do it. Thanks for this

[@adnanmadhoun8045]

This comment doesn't belong to this video, >> You saved me in my senior project last 6 months while I watched tutorial about web development ❤️❤️❤️❤️❤️

[@aniakubow]

Thanks so much for coding with me 💚

[@fourlokouva]

Love how you just get into it without any sitcomish-intros, just straight to the point. Your videos have given me the confidence to finally start applying to dev jobs. Thank you so much, Ania!

[@markharrisllb]

I am a boomer coding virgin except for a minute amount of C++ for my Arduino. No one has made sense to me before as there’s usually an assumption of a greater knowledge than I have. You are the first person who was coherent to me, every step explained and details of what is happening behind the scenes. Your beautiful diction helps a lot as well, thank you.

[@artopaivinen3233]

Thanks Ania! Tried already two small projects of yours, and I must say, that you're a teacher, that speaks to novices as well. The samples you made are easy to follow, and simple enough to figure it out, how the packages work with each others. Of course that also shows a great understanding, how to produce working program, but having success with working sample encourages to do more. Thank you with appreciations!

[@aniakubow]

This is so lovely for you to say and has made my day :) 🥰 thank you so much Arto!

[@artopaivinen3233]

@@aniakubow ... Sorry - forgot to wish Merry Xmas! ... and now I do. Happy New Year goes with the same package ;-)

[@metatronicx]

Clear and precise english. This woman explains well all topic she talks about. Thank you for being a content creator. It fits you so well

[@theshazman]

Would love a follow up to this with more advanced scraping such as a page that is behind a login wall, or something that requires a query to be filled before getting the final web content. Thanks heaps!

[@katykarry2495]

Works purrfectly and super simple code to use and follow. Can you do a part 2 where you learn us to crawl() with this scraping code? I mean the entire site for an example

[@emmabecker6460]

Yes please Khaleesi of codes do this!

[@noahmichael89]

Very cool simple project! One thing I noticed is that you must have nodemon installed globally (unless it's packaged with a version of Node I don't have?). For anyone that doesn't, the "npm start" script won't work. Of course you can just install it locally into this project with "npm i nodemon" in the terminal.

[@TheJosephTodd]

My man, thank you! Had this exact problem and you solved it. :)

[@authoritycamper]

Thanks man! This is the command for a Mac: sudo npm install -g nodemon

[@kenfoli330]

thanks alot man

[@maurov6861]

well done bro! installed on the project ;)

[@KingstonFortune]

This was a very good intro video to web scraping, thanks! As a quick tip, when running the `npm init` command you can just append `-y` at the end so it becomes `npm init -y`, and then it will proceed to skip the checks and create the package.json file without you having to press enter several times.

[@lilianamilano3707]

Thank you, Ania! It worked perfectly. I had no idea how to complete this task. You saved my day and gave me a lot of knowledge and fun too. I send you love from Venezuela, you are a genius! ♥

[@koenslootweg5882]

Just had a web scraper project on work last week so used your idea. Thanx Ania. had some problems with nademon... But just installed nodemon wit npm i nodemon

[@aniakubow]

Amazing!!! I’m glad this video helped and you managed to get nodemon installed :) thanks for leaving your advice for others too!

[@xxxbrenorxxx]

this channel is amazing!!!! study coding and simping at the same time

[@rinn1944]

Wow! You explained this so well and left nothing vague. A lot of other tutorials leave out so many parts, assuming everyone knows what they're telling. It's the first time I've come across your channel, and you just gained a subscriber. Thanks a lot!

[@syntaxed4365]

This was a fantastic tutorial, and as someone that is a lover of JS, it's nice to see this approach. I just have to say, I know that they are no longer required, but my brain just gets a completely unnecessary comfort out of using semicolons 😅 even though I can definitely agree that it looks cleaner without. I still have the desire to go through and add them everywhere! haha

[@aniakubow]

Thank you so much!! I really appreciate your comment 💚. Haha yes , I’m team no semi colons for my projects on here haha. I used to use them at work, but even then an extension added them in for me 😛

[@Ibloop]

hahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahahah >=[

[@jalapenogaming9740]

hey just here to say "thank you" !! you put a lot of effort into these videos i hope your channel grows fast and get to the top 🙌

[@glowiever]

I am assigned as data engineer and really need to scrape some data from a marketplace. this is a blessing!

[@galactic_4k]

Crazy, I just happened to need do something similar for a project and here you are uploading a video that helps a ton! Thanks!

[@Lucas_Gamaleri]

The mere fact that you have a couple longboards behind you convinced me to subscribe and like this video. You're a cool coder 🌝

[@elingrome5853]

🤮

[@lifelessalarm]

Thanks for the tutorial =) As a few others have said, an alternative option might be to use puppeteer. It has a nice syntax and is very flexible. You can simulate natural browsing using simulated click events and run additional commands such as taking screenshots of html content based on css classes.

[@LetsScrapeData]

yes, I design a template in 4 minutes to scrape title and url using a tool based on puppeteer: kzbin.info/www/bejne/qHOYc3udZb2sgdU

[@RealElites]

Learning from Ania's style of teaching is easy and relieving. Gonna binge this stuff

[@suavebob]

Not sure why this channel/video showed up in my feed but most definitely this lady is a damn reasonable POA

[@emilyreese9551]

This is a wonderful explanation of every line of code. I've learned code through online resources, mentors and college. Some of this stuff I knew I had to do but I did not know why. Thank you!

[@Cyapow]

Damn. Wish there were videos to follow like this 20+ years ago when I was starting out lol. You make it very easy to understand

[@jeevanjotsinghnarula2453]

Seeing couple of software out there that helps people scrape data from the website. It's great to watch this video so I can be able to do the same by just learning programming from best people over on youtube.

[@kamertonaudiophileplayer847]

You provide an amazing technique for web scrapping. It would be good if you explain also how to manage sites using an anti scrapping technique, for example instead of just populate a text directly, they wrap every character in a different tag or even replace some digits with a similar text characters and so on.

[@aminagha3206]

I am really proud that you are my teacher, I learned a lot from you, what is really interesting is that you are beautiful in appearance and also you have a beautiful mind you can explain the information in the simplest ways. Eagerly waiting for your new one🌹

[@networkserpent5155]

I wanted to learn the power of web scraping. This video shows just that! Thank you Ania

[@vskiy26]

Thank you for such awesome tutorial. Dziękuję Ci.

[@imagineaddis9050]

That is why learning docker was important for me. We can package our program to work on any machine without worrying about breaking changes from node or express. Thank you for the nvm tip!

[@havocblast8737]

yes docker is a life saver especially when you work on both mac and windows

[@jacksonaguilar1998]

I can listen to you all day. I hope you were my web dev lecturer.

[@AlcantaraMC]

sister used 70% of the vid to set up everything huhu. anyways, it works at the end, and its what we came here for. Thanks!

[@vinitmanerikar5444]

I have subscribed to many programming channels, however you are one of those who really add value..keep it up

[@edwarddejong8025]

Wonderfully clear exposition. A very effective teacher.

[@Shafty17]

Big thanks, Ania! With your help i managed to write a webscrapper to parse through all private repositories in my organization (had to add authorization as well), "read" it's package.json file, save it to a file, then run another conversion script that forms array of strings in certain way & then save it to .csv file, to be able to create a pivot table & analyze tech stack of our product :)

[@federicomontesdeoca8977]

Hello Ania, thank you for the quality of your videos ! I have almost 4 years of experience as a js dev, but still learning with your content. Just one question, why you install express and turn the app into a server, since you only need the http client (axios) in order to scrap the page?

[@kuldeepsharma7499]

Same question

[@PraneethParsharouthu]

same ques

[@lifeoffreelance9131]

Hi, this is my first time here, and i really love the way you teach. thank you for your very informative tutorial.

[@KacperSieradziński]

This is great that you are going through the doc during video :-) It is great because you teach people how to read the doc... I do the same on my Python videos :-) Pozdrawiam! :)

[@lloyd27z]

Everything you explained is quite clear and simple .So very helpful to learn.Thank you 😘

[@aniakubow]

Thanks so much Sue!!! It’s great to hear this feedback as I’m always trying to improve 😄💚

[@lloyd27z]

@@aniakubow looking forward to your new tutorial!

[@RealJustinEstrada]

Your tutorials are very consice and to the point. Its greatly appreciated.

[@fiveminutesvideo]

Hi Ania, love the videos! Would love one on setting-up a web scraper to scrape every minute or so in a Litespeed server. Keep up the great work.

[@maxmaksum4673]

Ania....it is just awesome...someday if you have holiday to Bali I will treat you and show you aound

[@MartinJaszczuk]

Great video. I love the longboard on the wall. Thanks for posting!!!

[@JoseHernandez-bg6zm]

Loving the bit more "fullstack-oriented" content! Actually have a project in mind where I could apply this perfectly. Thanks for the inspiration! ;)

[@CH3RRYxB0MBx]

I've been waiting for a tutorial like this! Thanks Ania!

[@rukkychew]

Thanks for the intro. After that I was able to move around and play around to build a scraper for blogspot

[@vancouverrrr]

You have helped changed the lives of many, thank you for sharing your priceless knowledge

[@saudahmed2436]

Thanks Ania, had a little bit issues but after i got nodemon package installed and realized i was missing a . for calling the class it worked like a charm :). One thing to note i think is that windows doesnt have a NVM, so i had to als install one of those just so i could make sure i am using same version as you

[@JordiVicensFarrus]

This is great Ania, thanks for your video! BTW just as an idea, would be great to have a second and a more advanced part, that shows how to do it in a site like Linkedin that requires Login or a site that has dynamic ids and classes. Cheerios!

[@DKSorg]

Awesome Video. Clear Precise and great Audio, easy to follow and listen to.

[@hestJaevel]

Great video! Only feedback is: Use chapters so we can skip the steps to install node if we already have it for example :)

[@romimaximus]

at 17:15 mins, i'm surprised, that you didnt get a "Cors" Error !!.... but great tutorial !!... thank you very much for sharing it !! 😃👍

[@Sloth740]

Thanks for the video! I had some issues with the "start" script on Windows but found that "node index.js" not "nodemon index.js" worked for me.

[@ksivasuriyaprakash9976]

Thanks for update my knowledge on npm. video was short and sweet.

[@shahedtheboss]

Thank you so much i needed to learn how to build a web scrapper You are the best❤️

[@zaliblk_media]

Great tutorial. Very well spoken. Very well communicated. Great Job Ania.

[@matthewknobel6954]

Back in the day I had to build a scrapper by hand with only perl in a unix environment. It would start at a root, parse all the html, and then traverse to the sub pages. All as an intern at Xerox back in 94.

[@aashishsinghal7331]

Awesome Video. Please Do a Docker Begginer friendly project with react or MERN, I have been trying to learn that, With your videos it will be so much helpful.

[@xmachina47x]

I think some of us, senior developers, will follow because we would like to see a surfing tutorial. .:D ..featuring the Lara Croft voice. Great coding! ;)

[@tancodes]

I have already creating a project with use of web scra and django. But this will be very helpful to learn from you 🔥

[@alexf1082]

Which Python package are you using? I don't like Selenium and I'm with requests-html but is no longer developed. The base requests package don't execute JavaScript.

[@luisamilcartavares26]

Interesting, whenever I think of scraper I always think of Python (beautifulsoup). This is really a very good perspective. thanks for sharing.

[@Workshop385]

I don't know anything about programming, but I watched the clip to the end.

[@discreetninja]

Thanks Ania, you're a wonderful teacher. As a newbie, your lesson is as simple as it can be for me to understand. I even worked around it to try some other projects too that I can think off 🤠💕

[@brookestephen]

I built a hierarchical object model to hold the parsed HTML, made of nodes and parms, and a Trie to hold the key words for the parser. I wrote it in every language I've learned... Python, Java, C#, VB... You can create HTML with it too, and output to a string, so it's in every web server I build. There's several different search methods, so adding to the object model is fast and easy... no more counting

[@trafalgarlaw4778]

learning my first line of code because my teacher Ania is stop eating beautiful

[@benimoza9746]

Omg, idk what words to say, i love you, this is so easy to follow

[@sadique_x_]

Ania makes programming very interesting!

[@SurfingTheMentawais]

Thanks for the video. You are obviously an expert.

[@KDevGaming1921]

Thank you this is a great help, It was working. I just had an issue with nodemon and installed it in my machine. I am just encountering the PORT not showing up on my terminal. I have an older version of node js though. Haven't tried using nvm yet. Going to try and practice that tip you gave. This was really helpful.

[@kuoyulu6714]

Thanks, this is really clear and simple to follow!

[@brussells]

Really clear and concise tutorial. Have a couple questions please: 1. How can you then use that data? Would you export to a csv, or database, to then use either on your own custom webpage or app? 2. Does this method run risk of being blocked by the webserver? Or would that not be an issue, as long as you're not scraping the data consistently?

[@javierfrancia1938]

i will build anything that beautiful Ania tells me, web scraper, 20 room house, spaceship, time portal, anything!!!

[@TheDa6781]

I could listen to her say scraper all day long. The part of my brain that would enjoy it doesn't have that many lines of code either.

[@odeddugma9748]

Thanks Ania. You are a great teacher ☺️ You make it looks so easy.

[@hazemelbatawy1242]

thanks, Ania it is very good and working perfectly.

[@whalebird]

great video and nice accent! It was pleasant listening to this tutorial, I've subscribed : )

[@colephares4973]

I don't think it was mentioned but I had to install nodemon globally to get this to run as it wasn't already installed.

[@kar0ee0m]

I subscribed today because of my friend Hesham recommended you, so don't disappoint me

[@okidokiyowyow356]

Thank you so much. I always wanted to learn how to scrape the web I didn't know it's this easy.

[@FeedScrn]

Scrape it for what?.. if you don't mind me asking.

[@sabatauqeer1]

just what i needed i successfully got src but was stuck at chaining thank you

[@mrCetus]

Amazing tutorial ! Keep up the great work ! The web scraper would now enable us to develop alternatives for all applications using APIs and this gives us great confidence.

[@im_cloudy]

I don't exactly understand why do we need to start an express server if we're trying to scrap an existing website. Is this some condition to run nodemon?

[@aniakubow]

It’s for part 2 :) we are going to show the results in the browser

[@ipunghendimunandar7827]

@@aniakubow so it can be called at endpoint right?

[@desarrollofacil411]

I was wondering the same, probably it will use a front end to get the url and display the results !

[@margin1636]

short, concise and useful. THANK YOU

[@honeyverma2755]

I usually not prefer to watch foreigner channel videos bcz your accent sometimes bit difficult 😅to comprehend but for you I must say you have given a good explanation of web scraping with this simple example 👍🔥

[@andrasbradacs6016]

I was looking for a completely different thing, but you gave a good idea with this video! 🤗 Thanx! 😁

[@gabyph2013]

Excellent video Ania! your explication is very simple and easy to understand, thank you

[@FranciscoBourquin]

Pretty helpful and easy to follow step by step explanation. I'm fairly new to programming so thank you 💕

[@aislinnsaunders-wallace9772]

Just created a stock data web scraping app thanks to this video!!! Thank you so much! I feel like a real pro lol :)

[@aniakubow]

That’s amazing! Thank you so much for sharing this with me 😍

[@seniorqa_auto5367]

Great lesson! Special thanks for enlarging the screen with the code!

[@naimulhasan3631]

Hi @Ania, Can we do this avoiding the legal issues? Can you please share your experience around the legal sides of Web scraping? Also how the companies does Web scraping avoiding the legal consequences of Web scraping? Please share your valuable suggestions. Thanks.

[@marianneazzopardi974]

Love your content Ania! I'm currently doing a bootcamp and your videos are helping me through!! :D

[@PeterCooperUK]

There's an extension called "Selector Gadget" which lets you pick elements on a page and it'll work out the best CSS selector to reach those elements which is great for scraping like this.

[@zippytyro]

Love it, so simple and unique way of teaching. Love from India >3000

[@AMoktar]

Awesome thanks, please more of advance scrapping

[@JakeAndDaddy]

Did you need to use express? You didn’t serve anything. Clear and concise. Loved that you zoomed in on the code. I’ve told numerous others I can’t see their code on an iPad.

[@aniakubow]

Express is for part 2 :) (coming soon)

[@tarek_maza]

Congrats on the 200k subscribers

[@aniakubow]

Thank you Tarek!

[@johnwayne8059]

Hi Ania, thanks for all the amazing content!👍👍👍 Your channel is truly helpful! You're simply great! 😉😇

[@thecoderabbi]

Subscribed! cause your contents are really great!

[@freedoom4090]

really nice! I did some of it in python, but it's kind of tricky sometimes, once many websites uses mechanisms to block selenium, scrapy and bsoup... I'll definitly try this one. Thanks for sharing! Always great content❤