Web Scraping with Python - How to handle pagination

[@oneashen4250]

Love this series man. I really hope for the advanced series too. Thank you for sharing!!!

[@JohnWatsonRooney]

thank you very much, very kind!

[@justwanttogroupdevstufff-yy7yj]

same here

[@Levy957]

videos everyday?? oh man, thank you for your time!!!

[@JohnWatsonRooney]

thanks!

[@deeperblue77]

Really valuable for all. Especially when new to this topic.

[@MrBenStringer]

Absolute legend. Amazing content. Learning a tonne, thanks dude 🙏.

[@Nas_Vinspired]

Great series! Thank you tons, man.

[@thebuggser2752]

Great presentation! Neat use of Python’s yield.

[@AmodeusR]

The next step is async scraping now 👀

[@sifar786]

Maybe if you could show how to pull all pages by handling how to bypass rate limit & ip blocking using rotating ip/ user-agent etc, then it becomes interesting! Hope you add such videos to this playlist.

[@AliceShisori]

thank you for this series, I think you should structure your future videos like this too. so maybe complex ideas/projects will be displayed better. you got a course or something on udemy? I'd love to buy it both to learn from you and to support you a bit to show my gradtitude. I don't have a visa or credit card so I can't thank you on youtube!

[@itumelengmadumo2925]

How would ou go about a webscraper that monitors changes to a website and notifies you ?

[@muhammedjaved4322]

Your videos are always amazing love your way of teaching can you please make video one google map contects scraping

[@danlee1027]

Great video as usual John. Per your other videos, would finding out max page count be alternate way for pagination stop condition versus checking for not 200 OK http response code? I like how you showed this option though. Thanks.

[@JohnWatsonRooney]

Yes I have done it that way before, sometimes there's justa "next page" button so you don't always know but certainly an option!

[@zakariaboulouarde4591]

Thaaaank you so much, veeeeery helpful 🙏🏾🙏🏾. You're the best. Do you have any recommandation where we can host like this script as an api with fastapi framework or flask?

[@JohnWatsonRooney]

There are free places but I generally use digital ocean - they have an app deployment service which i use. I also heard good things about railway

[@zakariaboulouarde4591]

@@JohnWatsonRooney Thaaaank you so much for your help and time 🙏🏾🙏🏾

[@benthinker]

THANK YOU!

[@SivaSakthiRajagopal]

Can you make a video to scrap the data from tripadvisor restuarant ?Like a big website

[@Fabricio-mq2uk]

John, could you tell me why httpx works with some urls and not with others?

[@Антмара]

Hello John. Thanks for your videos. I’m learning scraping and recently saw one order on freelance, I decided to complete it for myself (to test my knowledge). The problem with this task is that when there is more than one page in a category, the site only returns data from the first page. 72 products are posted on two pages, and when you collect information from two pages, you get 36 products that are duplicated. I think the site has parsing protection. but how to get around it? I use a random proxy and user agent. What do you think about this? Can you give me your hint, what is the matter here and how to solve this problem.

[@Omarwaqar-pt7wf]

Would love to see advanced web scraping with puppetier

[@Omarwaqar-pt7wf]

If we scrape a website let's say every hour generally speaking is there a chance that we'll get our IP blocked ?

[@JohnWatsonRooney]

Depends on a lot but if it’s smaller amounts of requests you should be ok

[@KontrolStyle]

Thanks for lesson. I keep getting "NoneType" error -- "AttributeError: 'NoneType' object has no attribute 'text'" - on 22 in video - but it still runs through with the code. if I just keep hitting continue. 😄

[@samoylov1973]

Following this tutorial and creating new scraping projects based on new knowledge. Can't figure out yet, how to get the actual html links. Say there's a code, that looks something like: ... txt. How to get this "/art/7/" part? I can get the 'txt' part from the a-link tag, but not the actual link, that I would like to follow later. Please, help.

[@JohnWatsonRooney]

instead of calling ".text()" call ".attributes["href"]" and it will get it

[@samoylov1973]

Thank you!@@JohnWatsonRooney

[@vinodbabu2965]

can you make a video on how to use neovim

[@JohnWatsonRooney]

Sure I can

[@WhiteFontStudios]

REI Shop: "Why is our conversion rate 100,000x lower on Camping and Hike Deals??"

[@juampivitalevi9611]

genius!!😁😁

[@mecrayavcin]

Can we scrape Java ScriptED sites with HTTPX and SELECTOLAX?

[@JohnWatsonRooney]

no you'll need something to render the JS, a browser, or you can look to find the sites API and see if you can use that

[@rohitlekhrajani6217]

@@JohnWatsonRooney does Playwright seem like a good choice?

[@JohnWatsonRooney]

@@rohitlekhrajani6217 yes it is, i've used it a lot and rate it highly

[@umerjavaid786]

I am learning alot John But i would recommend to make it more advanced level i had texted u at twitter too.. it would be of a great help if you pleaseeeee make a complete series related to scraping explain each n every aspect used in modern day scraping

[@umerjavaid786]

I had seen alot of tutorials but you are just beyong someone can even imagine how good you...i really want to appreciate you but i would say please make a complete series/playlist where you can start spreading knowledge from basic 1st step to the highest last step scraping diff sites n all more power to uh John ❤

[@DreamsAPI]

Pretty cool, can you do a video on scraping openapi specs from a website, if you have already can you post the link to the video? Thank you for sharing your knowledge.

[@usermae1407]

How the fuck can I do this to extract text like business titles, addresses and phone numbers?

[@Lukrafiveman]

this is for beginners? Imagine what you gotta do when youre advanced

[@bakasenpaidesu]

.......🎉... .