Python and Requests-HTML - Web Scraping Dynamic Content from JavaScript applications

In this video, we'll learn how to scrape content that is NOT present in initial page loads, but instead is loaded dynamically by JavaScript.
This is a common problem with scraping the modern web: the initial response contains minimal HTML and a SPA-based JavaScript app (React, Vue, Angular, etc). The data that we want to scrape on the page is therefore not present, but is rendered later via API calls from the SPA application.
We will look at how we can use requests-html to solve this issue in Python when scraping such sites. We'll also look at using this with BeautifulSoup in order to find data on the page.
This video makes use of the following sample website (a React application):
react-amazon-bestsellers-book...
📌 𝗖𝗵𝗮𝗽𝘁𝗲𝗿𝘀:
00:00 Intro
02:15 Sending GET request using Python requests library
04:00 Finding objects with BeautifulSoup
05:15 Installing requests-html
06:38 Executing JavaScript on page using requests-html
☕️ 𝗕𝘂𝘆 𝗺𝗲 𝗮 𝗰𝗼𝗳𝗳𝗲𝗲:
To support the channel and encourage new videos, please consider buying me a coffee here:
ko-fi.com/bugbytes
𝗦𝗼𝗰𝗶𝗮𝗹 𝗠𝗲𝗱𝗶𝗮:
📖 Blog: bugbytes.io/posts/
👾 Github: github.com/bugbytes-io/
🐦 Twitter: / bugbytesio
📚 𝗙𝘂𝗿𝘁𝗵𝗲𝗿 𝗿𝗲𝗮𝗱𝗶𝗻𝗴 𝗮𝗻𝗱 𝗶𝗻𝗳𝗼𝗿𝗺𝗮𝘁𝗶𝗼𝗻:
requests-html: pypi.org/project/requests-html/
BeautifulSoup: beautiful-soup-4.readthedocs....
requests: pypi.org/project/requests/
Sample website: react-amazon-bestsellers-book...
#python #webscraping #datascience

[@serdarfidan3451]

Hi, When I execute the response.html.render() function, I can see the terminal have downloaded the Chromium but it also throws the error down below: File "C:\Users\sfida\OneDrive\Masaüstü\python\pythonn\venv\Lib\site-packages\pyppeteer\launcher.py", line 227, in get_ws_endpoint raise BrowserError('Browser closed unexpectedly: ') pyppeteer.errors.BrowserError: Browser closed unexpectedly: Can you please help me out? There are no valid solutions on the web and I am using Windows at the moment. Thanks,

[@ahassan7270]

Thank you SO MUCH for your WONDERFUL explanations. You are really GREAT in communicating the ideas in a very clear and simple way.

[@bugbytes3923]

Thanks very much, that's great to hear!

[@GuilhermeBreda]

Thank you! Your tutorial really helped me in a web scrapping project.

[@bugbytes3923]

Glad to hear it helped! Thanks for commenting.

[@fernandtape9363]

Great content as always. Thanks.

[@bugbytes3923]

Thank you, as always much appreciated!

[@user-sl9fg8rl4j]

Wow, your explanations are absolutely fantastic! I really appreciate how you make complex things so easy to understand. Your content is always top-notch and has helped me out a lot. This video actually solved a problem I was stuck on. Thank you so much for all your hard work!

[@bugbytes3923]

Thanks a lot - amazing to hear that! Cheers.

[@RichIrl]

This was great! Thanks!

[@bugbytes3923]

Thanks for watching!

[@valentino7057]

I would love to watch a series of this! Talking about SPA's, how can we achieve SPA in Django? A series with Django and React/any SPAs would be great. 😃

[@bugbytes3923]

Thank you! I'll be doing stuff with React/Vue etc in future, for sure.

[@willysnowman]

Thank you! This worked in colab: from bs4 import BeautifulSoup from requests_html import HTMLSession url = 'URL' from requests_html import AsyncHTMLSession asession = AsyncHTMLSession() response = await asession.get(url) await response.html.arender() resp=response.html.raw_html print(response.status_code) # print(response.html.html) # print(response.html.find('article.book')) soup = BeautifulSoup(response.html.html, 'html.parser') books = soup.find_all('article', class_= 'book') for book in books: print(book.find('h2').text)

[@bugbytes3923]

Awesome!

[@abderrazzakaljalil1375]

Good job, keep going on 👍

[@bugbytes3923]

Thanks a lot!

[@ResilientFighter]

this solved my problem. thank you

[@bugbytes3923]

Glad to hear it, thanks for the comment!

[@gerhardspitzlsperger1567]

Very helpful

[@bugbytes3923]

Thank you Gerhard!

[@SOHAILKHAN-iu8fu]

Great content

[@bugbytes3923]

Thanks!

[@dobcs3236]

I do not know how to thank you, if there were a million likes, I would have done for you ... Thank you, thank you

[@bugbytes3923]

You are welcome! Thanks a lot for watching!

[@selvas5043]

Wow ...Thanks a lot.

[@bugbytes3923]

Thanks for watching!

[@ffgaming-fe3cx]

tons of thanks

[@bugbytes3923]

Thanks for watching!

[@sasanandeh3653]

How can i Handel button like add to cart?

[@Anu_was_here]

Not sure if you still respond on this old video, But i have a question: What if i have this "root", but it's a "tooltip-root"; that doesn't get filled with html unless I hover over the component. Noting that: it works without internet (if page is just loaded then i disconnect wifi, i can hover and see all contents). Would this library help me? (I went selenium route, but it's too cumbersome and slow with issues over time).

[@user-rj6hv4zf6w]

Thank you for the good content, is there a possibility to have a serie about how to use tasks on django (schdule tasks to excute on background) like doing a check every midnight...

[@andreaszweili8593]

Great video, Selenium would certainly be interesting but for my use cases I reckon this would most likely already be enough. BTW. how do deal with packages that haven't seen updates in a few years? The latest commit from requests-html is three years old at this point.

[@bugbytes3923]

Thanks! I agree, usually requests-html does the job unless we need to perform actions such as clicking elements to load content. Good point on the lack of updates. At the moment, I think the library continues to do its basic job well, so I'm happy enough to keep using it until a newer and better-maintained alternative comes along. There might already be one out there - if anyone is aware of one, please let me know!

[@frameff9073]

thank

[@bugbytes3923]

You're welcome!

[@SugengWahyudi]

Please add website that use login page and javacript and csrf token

[@Wesley9xd]

Not working in Angular applications.

[@rsmehta207]

Is it possible for you to help me?

[@brianaragon1641]

having some many porblems with libraries, i think many of them at this date, all changed in some way...

[@bugbytes3923]

Thanks for the comment. Maybe the video needs an update.

[@nnaemekacephas5931]

Did you make a video on how to use selenium?

[@bugbytes3923]

Not yet - would you like to see something with Selenium?

[@nnaemekacephas5931]

@@bugbytes3923 yes, that would be great