If you're web scraping, don't make these mistakes

Get $10 credit to try Bright Data: brdta.com/thepycoach_sb
In this video, we'll see some common mistakes to avoid and tips to follow when web scraping.
- Best web scraping libraries 2024: • What's the best Python...
- Command: exe_path --remote-debugging-port=9222 --user-data-dir="new_profile_path"
🔥 Best Selling Web Scraping Course in Python: www.udemy.com/course/web-scra...
My Courses
==========
🔥 Join My Automation Course in Python: www.udemy.com/course/automate...
🔥 Join My Python for Data Science Bootcamp: www.udemy.com/course/python-f...
💰 Make money by writing about AI, programming, data science or tech: thepycoach.teachable.com/p/me...
Support My Work
==============
💵 PayPal: www.paypal.com/donate/?hosted...
Content
0:00 Don't scrape the HTML doc (if there's a hidden API)
3:06 Don't send too many requests (or you'll get blocked)
5:37 Expect the unexpected
7:48 Don't stick to one scraping tool
9:55 Bonus - Workaround to login to websites
Disclaimer: This video was sponsored by Bright Data

[@KapuasInfo]

Thank you for giving me more knowledge on how to do the scraping

[@rogerleeaustin]

Thanks for helping us make fewer scraping mistakes. This should help me, especially captcha solving.

[@creativetech-hub]

I only knew selenium, thank you so much for charing this!

[@mysticpixai]

Thank you, @ThePyCoach! This video was incredibly informative and helpful.

[@GH-so4io]

Great tips! This will be useful for me building up my knowledge base via page scraping.

[@catris]

Wow these are some really great tips. Thank you!

[@carlpatterson9754]

Its great, provides an easy way to scrape. Thanks.

[@carlosfranco7788]

Many thanks for your wise recommendations to achieve a better, up to date web scrapping. Sometimes we are note fully aware on what's really important.

[@nicholasbimpong1537]

Good , helps me know more about Web scraping

[@dagosto]

Now I was able to make it work! Thanks for the content!

[@briangman3]

This bightdata is pretty cool, captcha are a big road block for me

[@prakashkaramchandani2413]

Very useful. Great tips for a newbie

[@LifeConqueror]

Thanks for this content, it's been a long time I'm experimenting on scraping and you helped me a lot!🎉

[@burakurun]

Great tips!

[@elmarvaher5017]

Good stuff. I have done some selenium and pandas to convert data on a spreadsheet to fill out on line forms. It took tweaking delay time to make it work properly. Thanks.

[@darrellsu1789]

AI has helped me a lot with summarising long-form writing/content especially with ChatGPT, ChatPDF etc. Thanks for this video too!

[@randybrown4649]

Great explanation about web scraping, look forward to learning more?

[@TonyGrantIM]

Hi, thanks for those tips, I have done quite a bit of scraping in the hsort time I have been using python. I think I mostly used beautiful soup and scrapy, but I havent got too deep really. In the most part I have been working with APIs, wich I personally find more reliable, if one exists of course. I think the frustration during those early days is lack of information out there. I find API documentation mostly awful. One little tip i did find helpful when researching api output etc is to check the pretty code option down at the bottom of the page. it makes it so much more readable and easier to copy into excel etc. Anyway, I do appreciate the content you put out there, it opens up new thinking, so thanks...

[@ulanonbergenov8063]

Nice tips for web scraping

[@bmorehead]

Good information

[@tribhuvanmishra7865]

I have worked in AI especially computer vision and also have done a project on web scrapping and deployed it. AI for me is a fun hobby

[@samanthamp3]

I know nothing about scraping but following you due to the AI content, and happy to learn from you!

[@SergioParreira]

Nice hints. I've only done very basic sraping with beatifulSoup, as part of some course exercises. I wanted to run a few things on a forum I was member but still struggled on how to authenticate correctly.

[@highspeedhomer]

Thanks for the scraping content and keeping me up on AI

[@BOGDES83]

Thanks! Could you do a video tutorial with the Playwright? Are you familiar with this framework?