The Biggest Issues I've Faced Web Scraping (and how to fix them)

Рет қаралды 23,034

Күн бұрын

Try out Bright Data and get $15 credit for your projects! brdta.com/fknight
0:00 Problems I face web scraping
1:03 Web Scraping Basics Overview
4:38 Handling Complex Web Technologies
6:24 Script Optimization + Error Handling + Adaptive Algorithms
8:23 AI-Driven Proxy Management, Anonymity, and Intelligent Rate Limiting
10:23 How to Handle Extracted Data
12:22 Ethical AI and Legal Compliance
14:15 Thanks for Watching!
If you're a developer, sign up to my free newsletter Dev Notes 👉 www.devnotesdaily.com/
If you're a student, checkout my Notion template Studious: notionstudent.com
Don't know why you'd want to follow me on other socials. I don't even post. But here you go.
🐱‍🚀 GitHub: github.com/forrestknight
🐦 Twitter: / forrestpknight
💼 LinkedIn: / forrestpknight
📸 Instagram: / forrestpknight

Пікірлер: 49

@delsix1222 Ай бұрын

interesting timing to see this video, literally the day after I completed my first full-stack application which literally revolves around web-scraping :D

@flipygmd Ай бұрын

You're the next Mark Zuckerberg

@Noumaan_Ahamed Ай бұрын

How do you web scrape secure website?

@xlafxx Ай бұрын

I remember starting to watch your videos when I was entering computer science Ba, and as a 28 year old 1 semester left to graduate, you’re still uploading good content that’s unique. Never get tired of your vids , keep it up brother . I’m also concerned with the job market , can you make a vid about new grad Cs students ? For example seems almost every job wants front end or something and my school never taught any of it

@mrrobot-mn6re Ай бұрын

You want to get a job from what your school taught you? You are in for a ride brother. Tech is about your own research and self learning, every fucking day.I pity people that majored in CS because they heard about a programmer earning 6figs

@doublesushi5990 Ай бұрын

such a chill vid

@EduardoEscarez Ай бұрын

AFAIK the button highlighting is a feature based on video subtitles, including those generated automatically, but still somewhat random. I didn't catch those because I was already subscribed and like the video a moment before you said it.

@v1d300 Ай бұрын

I don't think its a video subtitles feature. It just happens randomly in my experience. The thumb up button shakes and subscribe highlights. Didn't happen for me on this video though :(

@danielabraham3022 Ай бұрын

To be honest, i subscribed because the button lit up. Also, I love your content.

@redbill5197 Ай бұрын

Thank you for the amazing video! Much appreciated as a young web developer. By the way, none of the buttons lit up or did any animations... I am a subscriber, so I don't know if that's why. Peace!!!

@beaconxy 27 күн бұрын

It actually didn't.

@dalar2 Ай бұрын

I used to web scrape all the time, but stupid js frameworks obsfucated css class names has made it very difficutlt.

@sumukh007 Ай бұрын

The JD bottle in the background 😉

@tomasemilio Ай бұрын

Boom. Thanks

@xdcountry Ай бұрын

This guy gets it-I’ve been there. I can’t wait to make this all an easy ass python plugin

@Cryogenics12 Ай бұрын

Hi Forrest. I was wondering how you still feel about AI and the future of software engineering. With chat GPT out for over a year now, have your views changed much? Maybe a good topic for another vid.

@johnknox4293 Ай бұрын

interesting....thanks man

@ramelox Ай бұрын

When I see brightdata sponsorship, I instantly stop watching. Paying to brightdata is not a webscraping skill.

@zeddscarlxrd4331 Ай бұрын

Did u know how to bypass cloudflare or captcha without bright data?

@ZacMagee Ай бұрын

Some people 😂 That's like saying. "Oh well, these stupid people who drive cars, why would they do that when we still have horses?"

@vasyavasin7364 Ай бұрын

@@ZacMagee why should I pay it if I can do it free?😂

@vasyavasin7364 Ай бұрын

@@zeddscarlxrd4331 How to bypass cloudflare you can find easy.

@Ohiostategenerationx 26 күн бұрын

@@vasyavasin7364do you still not need to scrap a bunch of proxies to use?

@carsonjamesiv2512 Ай бұрын

GOOD VIDEO🎉👍

@yafethtb Ай бұрын

Yeah. Scraping a dynamic website really makes me want to scream like Linus Torvalds to NVIDIA. And I also hate CloudFlare 😂

@brianmorin5547 13 күн бұрын

Is there a reason/advantage to using Bright Data's "scraping browser" product instead of integrating their proxy and IP rotation services into a script I'm running on my own server?

@v1d300 Ай бұрын

I am working on building a project that heavily requires scraping so I been doing a lot of research. And its really hard to find anything good that is not sponsored by brightdata. I get it, their marketing team has done a great job with tapping a perfect niche of creators who provide valuable information but this also creates a problem to ending up finding that almost each good resource is related to using brightdata and its not something I want to pay for when starting a hobby project. Anyway, this is a great video either way. I learned a lot of things I hadn't considered in my planning. Like the ETL(thats a new rabbit hole I need to dive into) or adaptive content extraction to account of layout changes. I was just assuming I will set up reporting to notify me when I start getting no content and then I will fix it. So thank you for that. Do you setup redis or something to make sure some requests are accessed from the cache of recently requested data than scraping again or accessing the db? is that necessary? And at what point should a webhook be setup and for what purpose exactly? Thank you

@olasunkanmioyetunji9254 20 күн бұрын

Can you recommend a course to learn web scraping. A course that taught the tool and techniques you mentioned and other concepts

@phethindabamkhwanazi3546 Ай бұрын

Hey, man do you have another channel where you teach live?????

@phethindabamkhwanazi3546 Ай бұрын

If you have provide the link, please so I start learning more.

@dmytro-skh 23 күн бұрын

this video is what I need. But whoaa so fast changes of screens with code... I'm too old at 35 to be able to push the pause button so fast 😅 Do you have some links with those hacks?

@V4rrow Ай бұрын

dude is literally gilfoyle from silicon valley(love your vids)

@parten2323 Ай бұрын

i wasn't looking for web scraping video but his face drew my attention, i was like wait this is Gilfoyle right😂❤...

@FFl1s Ай бұрын

@javancheongyujing2531 Ай бұрын

Is web scraping under data science or software engineering structure?

@realshiiiiiit8349 Ай бұрын

Damn this guy is cool

@JoaquimDornelles95 Ай бұрын

My fucking hero

@einekleineente1 Ай бұрын

are there vids of that ???

@francishubertovasquez2139 Ай бұрын

Speaking of Females, if Hitler's fuhrer have Magog carrier of motorized machine monsters then the Northern Magog have ice snow predominant in their place near Arctic circle, and ice surface can better conduct gases and science elements and compounds interaction which can attract those science things from everywhere, who between them is stronger except for the Super Magog Dark Matter? Will they suffice at full force during the final battle end times?