Let's Build a WebScraper in 15 Minutes with NextJS!

NextJS makes web scraping super simple. Let's dive into the easiest web scraping method, simply using the native fetch API and then turning that into HTML! All in all, building a web scraper can be done in minutes and be really helpful when there is no API you can get the data from.
Be careful with web scraping though, some websites do not allow it (especially if the scraped content is behind auth or even worse a paywall).
With that said, I wish you lots of fun implementing this into your own projects!

[@BRP-Moto-Tips]

I appreciate these kind of videos that aren't quite the bread and butter for web development but still provide insightful knowledge on some of those not mainly used topics. Thanks for your work!

[@markovvoz8849]

Can you also provide a GitHub to this code. Thanks

[@alberteinstein7231]

Love the consistency Josh, thanks for helping me learn with you.

[@joshtriedcoding]

Glad to hear even Einstein is learning with me. Cheers man

[@kwyjibu]

What should i do when deploying project. It gives localhost error, i changed it to my own domain using https etc. but still not working. Any idea?

[@bigc4290]

Do you have the Github for this? I'm getting a 404 at the beginning.

[@satyarajraghav661]

I was searching for this. Thanks alot.

[@OfficialLRY]

Could you explain why you fetch the downloads on the server and not on the client?

[@okage_]

he said he prefers doing it that way since doing it client side will mean that the speed relies on the clients computing power which isnt always ideal

[@OfficialLRY]

@@okage_ oh i missed that.. thanks for making it clear

[@codewithguillaume]

Tomorrow my friend, I'll complete this video course for sure :) Thank you !

[@bryson2662]

This is a great video, I've been wanting to try some web scraping. Question though, why use t3 if you're not going to use trpc? I think trpc would make this more clean.

[@TimStark]

can you give an example of how trpc can make it more clean?

[@santiskiffa]

Why do I get a cors error?

[@hamzahmd_]

Awesome video.

[@dluancr]

Thanks, that way is much better than use puppteer. My code always break when I tried use the puppeteer

[@okage_]

amazing

[@athwala]

I tried scrape data from public Facebook post but I couldn't. How can I do

[@atlantic_love]

That's way, way too much work just to do something that's very simple.

[@benoitgrasset]

totally useless, you can't scrap most of the sites with this technique, for example the SPAs

[@OverMartinez]

What happend whe the site is a SPA and the fetch didn't get the all html from the server? there is any way to get full page content?

[@doncoder-channel]

Good vide, do you have a repo for this?

[@developer_hadi]

Thanks bro I'll watch this after finish learning react and start learning next js

[@hrithikwins]

Great determination ❤

[@pookie1307]

thank you for this video ive been trying to figure out how to use next js with puppeteer for a week now this helped me

[@1000ylovers]

Really Love the fact that you consistently put out contents that are actually very worth watching!. Josh, I have a question regarding if this can also scrap meta data of the website like title, image, subtitle and so on.

[@joshtriedcoding]

Cheers. For those use cases using puppeteer (a browser controlled from the server) will be of much more use to you than plain old fetch. Way more possibilities

[@1000ylovers]

@@joshtriedcoding thanks for the rely. Just to be sure, you are saying scraping metadata of the websites is indeed possible and one of the features of puppeteer?

[@joshtriedcoding]

@@1000ylovers For sure! For example grabbing the page title is as easy as page.title(). You can get additional metadata via the CSS selectors such as head > meta[name='description']

[@sarimkhan2306]

I am having error while importing jsdom library , any idea ?

[@groverkennedy2049]

Probably because of Next.js 13 with the client and server side