Python Tutorial: Web Scraping with Requests-HTML

[@waichan4476]

I don't usually comment in fact never, but I just wanna say thanks! the content you produce is by far the best I have seen

[@coreyms]

Thank you!

[@eeshsingh3336]

I see Corey's video. I hit like. 2am notification squad. 🤣 Hi from India!

[@SahajOberoi]

🤘🤙🤙

[@compinerd732]

17:23 sugar right there, i saw so many people just saying okay now we copy this, and put a 2 behind everything. oh my gawd. thank you - great tutorial so far

[@hell1018]

46:32 Javascript rendering 49:42 Asynchronous requests

[@akuamtau]

thank you!

[@basseygodwin7384]

Thank yu

[@neoninsv]

I am a big fan of you showcasing what it looks like first, then breaking it down in detail.

[@ricksegalCanada]

12 minutes in, I can grab website information from this tutorial. Why is this a big deal? I know next to nothing about Python. Corey is high value in a very condensed time. Others would take hours to get to his 12-minute mark. Subscribed.

[@andrewmendela9065]

Corey, you are a wizard! I dont know how are you doing this, but by far your videos are the best educational programming material that i have ever seen, thank you! When i get my first job you will be the first person i pay

[@TedMaciag]

Hey Corey, BEST video for an older programmer to understand. Thanks very much!

[@kristianfjeldepedersen4675]

It is so satisfying how you always somehow cover the topics both me and the rest of the notification squad find most interesting. Keep up the excellent work, Corey.

[@edemaehiz]

Menh. woke up this morning with a desire to reinforce my knowledge on Beautiful soup and APIs. Your videos did it for me. In Bini, Nigeria we say "Uwese kakabor" meaning thank you so much

[@coreyms]

Thanks! Glad it helped

[@dimitriskatsoulis4986]

@@coreyms Man that's crazy to have viewers worldwide! Ehis from Nigeria, another comment above from India, you mister Schafer you are loved all around the world! Greetings from Greece : )

[@ericli9292]

What a great tutorial! I bet this is the first long tutorial that I ever watched nonstop.

[@coreycarter5668]

Hahaha! As soon as I see a notification that he has uploaded a video it’s like Christmas morning!

[@ahmed_ziada]

If I could give you a billion likes for this video I would. This is top quality content.

[@justinpopa9399]

You could not have timed this more perfectly for me. I was planning on working on figuring out how to do exactly this while relaxing this evening. Thank you!

[@archstampton5910]

Thanks Corey, I just finished the video. I appreciate the fact that took some time to go through some non-directly related topics (csv files , splitting links, etc ...) As soon as as I will be a bit more confortable on Python , I will read that American Doll Bed link of yours.

[@debbygram8153]

real heroes don't wear capes they teach like Corey Schafer. You literarily make programming simpler than A, B, C.

[@yubero2010]

Daaamn this is the greatest video I’ve ever seen about scrapping, nice I was looking for this kind of explanation for long time since I’m working on a project with python 3

[@godfreynolottyogwu8562]

The moment l see your notification ,l start dancing because it's always a hit,thumb up Boss.

[@benwalsh2825]

This is really top notch. Thanks so much for putting this together. Very well done!

[@coreyms]

Thanks! Glad you liked it!

[@speaktothepoint2108]

That’s fantastic. Corey makes it very simple to understand the complex topics.

[@delllatitude299]

man i need a button to give 1M likes at a time to your video. JUst amazing specially the last part. You clear my big big big big concept

[@denniskimani6810]

thanks for effort. your tutorials have been an important part of my journey as a programmer .

[@maciejZar]

Hi Corey, Great video! Thank you. About your question concerning “beauty” of scraped webpage, I think that if you do session = HTMLSession() r = session.get(url) print(r.text) not r.html you have in fact html but ordered somehow. Best

[@josephsowah1678]

Thanks Corey.. you're the best...love all your tutorials. Looking forward to your tuts on rest api in python.

[@t-dsai]

Hi Corey, Thank you very much for sharing your knowledge. Apart from your programming skills, your pedagogical skills are also of high order. I presume that you wanted to explain various aspects of splitting, and creating yt_link with string format. Otherwise, a shorter ways of getting the yt_link (from 29:55 to 34:45) could be the following yt_link = article.find('', first=True).attrs['src'].split('?')[0].replace('embed/', 'watch?v=') It splits the whole source at '?', and then replace the 'embed/' part of the string by 'watch?v', and directly returns the link.

[@LookNumber9]

Superb! That's just what I needed for my next little project. Thanks.

[@CompThatHouse]

Thankyou, Corey. Your explanations are always complete and very helpful!

[@teleleuinbedeleu]

Hi Corey, first of all, thank you, I just started to learn python and your videos are super helpful. I'll love to see in the future a Dash tutorial :)

[@HarmanHundal01]

Just a suggestion Corey. Can you please tag your videos 'Beginner', 'Intermediate', 'Advanced' for the benefit of noobs like me. Thanks already. Keep the awesome stuff coming.

[@dheerajkura5914]

Corey, you're a gift for the programmers. long live and spread the knowledge Can you please do the Vidoe's on OpenCV as well which is useful for Computer Vision

[@winglau7713]

A great video, everything is super clear, you are a gifted teacher, thx so much!

[@arcy2056]

You are the best, Corey 🥳

[@AlexBerkk]

Can we, please, have a tutorial on async/await and yield from? I kinda get it, but I really want to hear your explanation. Thank you so much for your vids!

[@ericklungle5715]

Great tutorials. Most people I have watched also will take requests. I have been trying to figure out how to change an element on a webpage and as yet have not seen anything very useful THAT WORKS!! It would be nice if someone would do a tutorial on how to do the following: change the following:   Single Draw to this: (Notice the value="" parameter)   Single Draw

[@whateverbefore]

I think I'll rewatch this gem some times....

[@gisleberge4363]

Very thorough and complete on the topic, thanks for educational video 🙂

[@vinayagarwal1623]

A video about dynamic web scraping with Selenium would be super helpful. I was having trouble with moving buttons on a website. Thanks in advance.

[@antonyalen2745]

Hi Corey, I really enjoy watching your tutorials. Please make a tutorial on Asynchronous running of code.

[@whilelab]

Just want to add that you can access a prettified version of the HTML using r.html.html BTW Great video.

[@alexgolotte8016]

tried but gives me errors File "./PYTHON/authWebsiteReqHTML.py", line 51, in responseGetTagInfo print(tagInfo.html.html) AttributeError: 'str' object has no attribute 'html' called my 'r' - reponse -> 'tagInfo'

[@davebeckham5429]

Excellent tutorial as always. - Many thanks.

[@sarunassavickas5351]

Very informative and useful tutorial. Thank you, Corey! Would love to see your approach on asyncio or multithreading :)

[@mytimeincloud5263]

Just Supported to show my gratitude.

[@coreyms]

Thanks so much!

[@nihalsharma567]

@@coreyms please make a playlist on AI

[@drinkingineasterneurope6947]

I had some issues with beautiful soup but with requests_html nothing. For me this module is the way to go. Thank you for showing this alternative it turns to be better!

[@DanielWeikert]

Once again great video. Thank you very much Corey. I don't know anyone else who is able to explain python that good. Could you consider to make a video on pathlib? Best regards

[@Seppiik]

Simply described to the point! Thanks

[@tiger12506]

This is really cool. I was looking for the ability to scrape a website and found requests_html. Quickly ran headlong into a wall as the site is a React.js site. :( Thought maybe I could find some information on performing clicks and such with requests_html, but looks like that is not possible. Your tutorial on the subject is great though. Really well thought out and explained, Great presentation!

[@roccococolombo2044]

Great videos and thanks for using large readable fonts.

[@carlosmatosfanpage2856]

How do I display results of scraping on my own website using Django.I want to make a website that compares prices of products across different websites but don’t know how to put the data which I have scraped onto my website. Thanks

[@ridrugo182]

On 29:36 why are the first two classes written after a dot "." but the class isn't?

[@hugogradvohl1549]

Hi Corey, I was hitting a wall with asynchrone programmation and the last part has help me a lot. Since "await" doesn't work with all functions and the methods name have been upgraded, I was looking for something consistent. Has often I end up finding the right way to use the functionality in your video. I know that you have many projects but asynchrone and thread programmation are a very interesting topic, if you run out of projects (not happening ^^) it would be great if you could make a light series of video (by the way I reduce my download time from 58 sec to 18sec by applying async functionality). Thank you!

[@cybergen2K]

Hi Corey, you're the teacher we've always needed :) Now for a creepy request: Would you ever go back to your OOP Tutorials from 2 years ago and change them at all?

[@abhishekvaish6042]

Awesome video and I would love to have a video on how to keep changing ip while requesting

[@anunayanu]

what is difference between using Beautifulsoup and Requests-HTML

[@KushChoudhary]

dynamic rendering is where problem starts when working with bs4 and requests. Requests-HTML has requests builtin with code rendering and parsing in it, what selenium can do can be done using this lib and this is cool!

[@SeamusHarper1234]

Also, you can use BeautifulSoup on HTML that is rendered by requests-html.

[@artabra1019]

@@KushChoudhary thats really great

[@Troglodyte2021]

Brilliant as usual! Salute!!!

[@michealhall7776]

Can you do a full video on asynchronous requests

[@earltan739]

Awesome!!!! Been looking for this man!

[@romankhripunov6550]

Как всегда круто! Самый качественный контент ) спасибо, бро!)

[@christianrusso5142]

Very clear and helpful, thank you!

[@farmakoxeris]

Great video buddy

[@husseinalahmad429]

Brilliant, thanks a lot Corey

[@kingstonpeng1076]

Forgot to mention: I was using VSCode, not PyCharm when I ran the following one line python script.

[@Dennis-McTatten]

top notch as usual thank you

[@prateeksarangi9187]

Wow detailed info !! Request to go for coroutines and asyncio and async await please

[@PawelVerma]

At 25:35 headline = article.find('.entry-title-link',first=True).text # where did this dot come from before entry-title-link? Why is this needed, did I miss some explanation?

[@slobodantajisic2762]

CSS class selectors are denoted by . , for example .somename, and CSS id selectors by #, for example #somename. At 13:45 you have match = html.find("#footer", first=True) . Look at this htmldog.com/guides/css/intermediate/classid/ for a start.

[@hsumerfarooq5474]

Please make scraping tutorials with scrapy and selenium too, BTW thanks for your efforts.

[@DI-xs3kh]

Hi @Corey, for your tutorial related to AsyncHTMLSession. I'm getting the "RuntimeError: This event loop is already running." I checked the documentation did not really see the reason for it. Could you please take a look if that is expected. I'm running in Windows 10. Python 3.10.

[@draco76xx]

Great video but would like to see more example on scrapping dynamic image etc using the render() function or something.

[@_ARIC_KAJI]

it's very useful thank you so much 💯

[@samirsarkar001]

You rock man 🤘🏻

[@compucademy]

I can't find an answer for this anywhere, maybe you can help. Is it still worth learning to use Beautiful Soup, or has Requests-HTML basically superseded it, even though not many people have caught up?

[@rahulsoni1969]

Sir can you please teach me how to use render() function properly.I am facing huge problem for scrapping data from a web which loads results dynamically using jacascript

[@stocksunlocked]

Great stuff. Quick question. I'm able to scrape links but when they output on the HTML page it's just the text, not the clickable hyperlink. Any ideas on how to fix this so I can have a clickable link?

[@stephenaborhey4214]

i really love this video @Corey Schafer but i would like to learn about using the api to scrap data from social media like Facebook, twitter and the rest so if you do a video about that will be appreciated thank you

[@chashmal10]

On the webpage/url that I call session.get(url) on, there is a javascript script, one thing this script does is send a request of its own, how can I capture the response to this request?

[@benedikt78]

I usually use Selenium if I want to scrape JavaScript content. When would you use Requests-HTML instead of Selenium? As far as I know, Requests-HTML uses headless Chromium to scrape JS data.

[@emasmach]

Nice video!

[@albertomedinarobredo]

Hi Corey, this explanation is great, as always! I'd love to see a more difficult scraping example. Are you planning on doing something like that? Or do you have any recommendations of what to read/watch? Thanks!!

[@enia1953]

Can you do a webscraping with python and scrappy and xpath and Hidden API.

[@sandeepvk]

Pls do try to scrape a public website using their api

[@curruption018]

Whenever I run .find(), the type thats returned is a list. For example the variable you have named "headline" would be a list. So I cant run .find() again. Also for some reason it's not recognizing .html as a method of the r object. I even explicitly declared the variable type but it still cannot see .html as a method from whatever session.get returns. Any suggestions?

[@paulseldn]

Hi Corey. Can you please tell us what IDE you are using. it is so nice to be able to read large fonts and also resize the console. I cannot do this in Atom

[@joy2000cyber]

How AsyncHTMLSession work with concurrent.futures? Don’t want to write a function for each thread.

[@sandeepvk]

Hi, When I try to run the code : from requests_html import HTML with open('simple.html') as html: source = html.read() print(source) I can still seen and html body generated in the console. Why do have i got to them move the source variable into another with "HTML(html = source) and print the result ? in both case I get the same result isn't it ?

[@michaelhrabe4139]

"source" variable in this case is just a text, though html = HTML(html=source) is an instance of requests_html.HTML class. You wouldn't be able to execute functions like find on "source" variable.

[@sandeepvk]

@@michaelhrabe4139 Dear Michael, thank you very much for taking time to answer. I understand now. thank you

[@im4485]

Hi Corey. How does one find an element by its attribute and not by using css selector?

[@varun83s]

Hi There, When I am doing response.html.render() I am loosing authentication. When I do response.html.find('div') I get all the desired results however as you mentioned to use just html.render before find, I am not able to hold authentication when chromium is working to get the data. Any clues how to resolve this. Any pointers is highly appreciated.

[@SeamusHarper1234]

Hi, what if the data that you are after only exists after some user interaction, e.g. clicking a button or triggering some other js event? Can you simulate this with requests-html?

[@kinjalvora256]

Hey Corey, Great tutorial I have a small doubt though articles = r.html.find('article') would this technically work for all websites? how to look for the information under inspect I was trying to use this to try and scrape a headline from BBC news and it does not seem to work I am not sure what information I should look at under inspect to make sure I have the right thing selected maybe you can help with that

[@barrentheart2680]

Awesome content! Can you do a tutorial about Scrapy?

[@manpaalsingh]

Can you use python to automate and export every iteration of a searchable or filterable website?

[@feather1x]

Hey corey, whats the difference between the video where you used beautiful soup to scrape information and this video?

[@unique1o1-g5h]

please do a detailed video on async

[@edemaehiz]

Hello Corey is it possible to create a spidering program using requests_html or do I need to use Scrapy for that

[@joelprestonsmith]

I'm learning a ton about webscraping from this tutorial, but I'm not able to run the code. Like many folks, I've got a few Python versions installed. I ran the code in the Thonny IDE, but I get a traceback on 'no requests_html module found.' Did some research on it, and discovered that requests_html is only supported on Python 3.6 (and my Thonny default was 3.7). I reset Thonny to run 3.6.5, but got the same error. Now I'm installing 3.6 to see if requests_html will be imported in that version. Anyone else see a similar issue with a traceback? What was your workaround?

[@farmakoxeris]

45:53 How can I get the number of the html links contained inside r.html.links?

[@slobodantajisic2762]

num_links = 0 for link in r.html.links: num_links += 1 print(num_links) or just print(len(r.html.links))

[@eddievuong]

Hi Corey, thanks for your video, it's really helpful. I want to ask if the website requires log-in to see the data, how can we do that? I see there's a way to do it with normal request library but found none with requests-html. Thanks

[@robcarreon5743]

Thank you for your videos! They are very well organized, easy to follow and extremely helpful! I followed through on the above video and got it working perfectly on a site that uses javascript. But I only got it to work when running via python command line or shell. When I put the code inside a very simple "hello world" django project, the .render() function causes a Thread Loop-1 error; if I comment out the r.html.render() line, I don't get the error, but the information I get back is incomplete. I searched all over for results-HTML and django and this error and couldn't find much on a cause/solution. Just curious if you've run into this and know why it doesn't work? Thanks again!

[@samirsarkar001]

I like your tutorial most. I just have one question. For parsing the article headline and summary you used ".entry-title-link" and & ".entry-content p" in find method but at the time of vid_src you used "" . So how we decide when to use . and when not ?

[@theglobalconflict6904]

do you have any series about asynchronous programming in python ???

[@ori61511]

31:00 why not just use find("embed/") then find("?") like this: url[url.find('embed/')+6:url.find('?')]

[@suthejganjam1395]

Hi Thanks for the video.Can we get access to DOM object using this plugin