Python and Requests-HTML - Web Scraping Dynamic Content from JavaScript applications

[@serdarfidan3451]

Hi, When I execute the response.html.render() function, I can see the terminal have downloaded the Chromium but it also throws the error down below: File "C:\Users\sfida\OneDrive\Masaüstü\python\pythonn\venv\Lib\site-packages\pyppeteer\launcher.py", line 227, in get_ws_endpoint raise BrowserError('Browser closed unexpectedly: ') pyppeteer.errors.BrowserError: Browser closed unexpectedly: Can you please help me out? There are no valid solutions on the web and I am using Windows at the moment. Thanks,

[@ahassan7270]

Thank you SO MUCH for your WONDERFUL explanations. You are really GREAT in communicating the ideas in a very clear and simple way.

[@bugbytes3923]

Thanks very much, that's great to hear!

[@GuilhermeBreda]

Thank you! Your tutorial really helped me in a web scrapping project.

[@bugbytes3923]

Glad to hear it helped! Thanks for commenting.

[@willysnowman]

Thank you! This worked in colab: from bs4 import BeautifulSoup from requests_html import HTMLSession url = 'URL' from requests_html import AsyncHTMLSession asession = AsyncHTMLSession() response = await asession.get(url) await response.html.arender() resp=response.html.raw_html print(response.status_code) # print(response.html.html) # print(response.html.find('article.book')) soup = BeautifulSoup(response.html.html, 'html.parser') books = soup.find_all('article', class_= 'book') for book in books: print(book.find('h2').text)

[@bugbytes3923]

Awesome!

[@adamcarvideos]

thank you sir

[@AleemAhmed-g3n]

Wow, your explanations are absolutely fantastic! I really appreciate how you make complex things so easy to understand. Your content is always top-notch and has helped me out a lot. This video actually solved a problem I was stuck on. Thank you so much for all your hard work!

[@bugbytes3923]

Thanks a lot - amazing to hear that! Cheers.

[@valentino7057]

I would love to watch a series of this! Talking about SPA's, how can we achieve SPA in Django? A series with Django and React/any SPAs would be great. 😃

[@bugbytes3923]

Thank you! I'll be doing stuff with React/Vue etc in future, for sure.

[@dobcs3236]

I do not know how to thank you, if there were a million likes, I would have done for you ... Thank you, thank you

[@bugbytes3923]

You are welcome! Thanks a lot for watching!

[@nox6687]

If you're running this in Jupyter notebooks, you need to use AsyncHTMLSession instead, so it would be: response = await asession.get(url) await response.html.arender() print(response.html.html)

[@rubenagurcia906]

what happend when tha data is load by a json and if you want to consumme it is block, it is posible to scrappe the page with react and data load with json?

[@fernandtape9363]

Great content as always. Thanks.

[@bugbytes3923]

Thank you, as always much appreciated!

[@muhammadnoman6205]

Superb as always..

[@bugbytes3923]

Thank you!

[@RichIrl]

This was great! Thanks!

[@bugbytes3923]

Thanks for watching!

[@abderrazzakaljalil1375]

Good job, keep going on 👍

[@bugbytes3923]

Thanks a lot!

[@sasanandeh3653]

How can i Handel button like add to cart?

[@MrRedStream]

Great and brief material. I've stucked after package instalation (succesful), but while executing a code get error (Python 3.10 I'm using VSC) from requests_html import HTMLSession NameError: name '_string' is not defined I suppose I'll try selenium instead.

[@Anu_was_here]

Not sure if you still respond on this old video, But i have a question: What if i have this "root", but it's a "tooltip-root"; that doesn't get filled with html unless I hover over the component. Noting that: it works without internet (if page is just loaded then i disconnect wifi, i can hover and see all contents). Would this library help me? (I went selenium route, but it's too cumbersome and slow with issues over time).

[@selvas5043]

Wow ...Thanks a lot.

[@bugbytes3923]

Thanks for watching!

[@ResilientFighter]

this solved my problem. thank you

[@bugbytes3923]

Glad to hear it, thanks for the comment!

[@SOHAILKHAN-iu8fu]

Great content

[@bugbytes3923]

Thanks!

[@andreaszweili8593]

Great video, Selenium would certainly be interesting but for my use cases I reckon this would most likely already be enough. BTW. how do deal with packages that haven't seen updates in a few years? The latest commit from requests-html is three years old at this point.

[@bugbytes3923]

Thanks! I agree, usually requests-html does the job unless we need to perform actions such as clicking elements to load content. Good point on the lack of updates. At the moment, I think the library continues to do its basic job well, so I'm happy enough to keep using it until a newer and better-maintained alternative comes along. There might already be one out there - if anyone is aware of one, please let me know!

[@gerhardspitzlsperger1567]

Very helpful

[@bugbytes3923]

Thank you Gerhard!

[@مكتبةرفوف]

Thank you for the good content, is there a possibility to have a serie about how to use tasks on django (schdule tasks to excute on background) like doing a check every midnight...

[@nnaemekacephas5931]

Did you make a video on how to use selenium?

[@bugbytes3923]

Not yet - would you like to see something with Selenium?

[@nnaemekacephas5931]

@@bugbytes3923 yes, that would be great

[@Wesley9xd]

Not working in Angular applications.

[@ffgaming-fe3cx]

tons of thanks

[@bugbytes3923]

Thanks for watching!

[@SugengWahyudi]

Please add website that use login page and javacript and csrf token

[@brianaragon1641]

having some many porblems with libraries, i think many of them at this date, all changed in some way...

[@bugbytes3923]

Thanks for the comment. Maybe the video needs an update.

[@rsmehta207]

Is it possible for you to help me?

[@frameff9073]

thank

[@bugbytes3923]

You're welcome!