Rendering Dynamic Pages 2! - Web Scraping ALL products with Python

Part 2!
Part 1 is here: • Render Dynamic Pages -...
Code: github.com/jhn...
In this video we clean up the script from part 1 by adding functions, and learn a way to deal with pagination. I run the completed script at the end and scrape all 796 products with info from the site.
-------------------------------------
twitter / jhnwr
code editor code.visualstu...
WSL2 (linux on windows) docs.microsoft...
-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
mouse amzn.to/2SH1ssK
27" monitor amzn.to/2GAH4r9
24" monitor (vertical) amzn.to/3jIFamt
dual monitor arm amzn.to/3lyFS6s
microphone amzn.to/36TbaAW
mic arm amzn.to/33NJI5v
audio interface amzn.to/2FlnfU0
keyboard amzn.to/2SKrjQA
lights amzn.to/2GN7INg
webcam amzn.to/2SJHopS
camera amzn.to/3iVIJol
gfx card amzn.to/2SKYraW
ssd amzn.to/3lAjMAy

[@engineerbaaniya4846]

Very helpful u have unique content for scraping dynamic websites please keep posting such videos

[@Semihszy]

i saw like that you scrolled down and the first page loaded new products so when you use this method you get everything on each page without the need to scroll down code? and I am really grateful for your work it helps a lot but the most important thing for me is that your videos are like my own notes I took in a class :) I have tabs saved full of your videos and I created a tabbook term for myself thanks a lot man 👍

[@sayyadsalman9132]

Great work bro! Keep it up. Thanks for such a quality content.

[@adnanzafar5385]

Great work, John... it's beneficial for me.

[@amineboutaghou4714]

Great video John as usual ! Many thanks 🙏🏼 Looks like HTMLSession is very similar to selenium isn’t it. Could you please explain when we should use one or the other and which one you prefer when scraping dynamic web pages ?

[@JohnWatsonRooney]

it is similar as in they both use a browser to render the page. Selenium allows us much more control of what the browser does, so consider if you need that or not for your project. If not then give requests-html a go!

[@amineboutaghou4714]

@@JohnWatsonRooney Many thanks John for your reply :) I will definitely give it a try ;)

[@rahulgurave6429]

Very Nice Video John. Thank You So Much..!! :)

[@saeeahmed5213]

thankssssssssssssss so much man! I have learned so much from your channel and you also made my work very easy I was stuck.......again man lots of love from my side♥♥♥♥

[@gitgosc7075]

great show!

[@adrianamejiaalegria2860]

How do you deal with the 403 error when trying to access all the links that you've stored in your first request? Thanks for this tutorial, John!

[@Neil4Speed]

All Done!! Awesome :)

[@sinamobasheri3632]

great content thanks 👌🏻🙏🏻

[@SunDevilThor]

Any reason as to why all the ratings would show up as a negative number in the CSV file?

[@elmo7409]

thank you very much ... I HAVE A PROBLEM WHEN I USE THIS ; i get this error RuntimeError: Cannot use HTMLSession within an existing event loop. Use AsyncHTMLSession instea

[@azaf]

Can you please teach us how to store this info in a mysql database? I would greatly appreciate it. I know there are videos online but I love the way you teach 😊.

[@JohnWatsonRooney]

Yes that’s a great idea I will work on a video that shows output to different formats

[@azaf]

@@JohnWatsonRooney Looking forward to it brother.

[@harshgupta-ds2cw]

Hey john i am trying to webscrape ott platforms (Hotstar). I am unable to scrape anything.

[@artabra1019]

what is difference of beautifulsoup to request_html ?

[@JohnWatsonRooney]

They both parse the information for us - generally I prefer beautifulsoup but requests_html is also very powerful, as it can work with xpaths and css selectors

[@mohfatkurrozi4069]

Awesome :-)

[@mohfatkurrozi4069]

I try to scrape any website but i was blocked identification robot.. How solve it john! I using request libs

[@mohfatkurrozi4069]

Error 403

[@hurstilthymy4943]

First! And thanks for the great content, John!

[@JohnWatsonRooney]

Thank you!

[@sinamobasheri3632]

👌🏻👌🏻👌🏻👌🏻👌🏻👌🏻👌🏻👌🏻👌🏻👌🏻

[@JohnWatsonRooney]

👌👌

[@aminaghasi611]

I wish you were my friend buddy:D

[@SabineWildemann]

i think i'm through 50% of your videos now.😇 Thanks a lot for this great content, i really like the way you teach the content and i'm learning a lot (just started with Python;-) Is there any combination of this video and kzbin.info/www/bejne/bpXVdnVma7adgtU (Async)? I tried to combine the code but failed. I have a site to crawl that seems to have a Tarpit operating....

[@JohnWatsonRooney]

hey thank you! very kind. I haven't done much async that uses a browser (like this render()) actually, I do believe it is possible though. Generally if i need to use a browser I am not worried about speed. Tarpit is an interesting one and something I've not seen a lot, but I believe it just keeps the connection open and doesn't respond if it thinks you are a bot. Interesting approach, i would just use a timeout and retry the connection

[@SabineWildemann]

@@JohnWatsonRooney Thanks, i'll try the timeout then. While reading through the comments i thought maybe an upvoting tool(/website would be great where people can leave topics and can vote for videos you could record.;-)

[@JohnWatsonRooney]

@@SabineWildemann that's a cool idea - my only worry would be managing it and trying to prevent abuse!

[@SabineWildemann]

@@JohnWatsonRooney I think it's worth a try.;-) I'm just thinking about how to best visualize the scraped data. Do you have any tool tip for that?

[@JohnWatsonRooney]

@@SabineWildemann I'd recommend Pandas for the data part and streamlit to turn your data into visual apps