Web Scraping with AIOHTTP and Python

[@Rasstag]

John, Great tutorial... many thanks... now I know how to juggle... ;) Wanted to pass on an observation... apparently, Windows, can be cranky with asyncio/aiohttp. Your example program throws a “RuntimeError: Event loop is closed ” error. However, adding towards the bottom: asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) … on top of pages = asyncio.run(main(urls)) … solves whatever 'Event loop' issues that were present.

[@rtxmax8223]

yes i was working on the async request and I got the similar error, the only way it works for me is if my program keeps on running. Since I am doing a constant request response after one ends, i dont get this error. but when I run my program for only one page to scrap then I get this error.

[@luxeave]

@Rasstag, dude ur a life saver. well done!

[@Valentin439]

Thank man!

[@efferington]

Hi John, thank you very much for this. Found this video while trying to figure out how to include an async AIOHTTP loop in some API processing script I'm writing and this was invaluable for figuring out how to structure the code.

[@andres777video]

No requests from me... just love your videos John! - Thanks for spending the time... I need this code to pull data for 188,600 items (each one is a web page...with 3 tables each) - UIPath would take about 32 days to complete. - asyncio + aiohttp should be much, much faster! Thanks for the tip Rasstag ( had the same issue in Windows)

[@JohnWatsonRooney]

thank you very much!

[@a_private_handle]

Great wallpaper from Firewatch

[@yacinehechmi6012]

Thank you sir!!

[@symbolminded5167]

Your video helped a lot, thank you

[@nsnilesh604]

Awesome way to get list of urls 👌😎

[@JohnWatsonRooney]

Thanks!

[@tubelessHuma]

Always trying new ways of scraping. Great 👍🌹

[@JohnWatsonRooney]

Thanks!

[@fuad471]

Thank you it was really helpful to grasp the asyncio concept

[@rrahll]

thanks for the tutorial!

[@МаксимМельников-ж4у]

Really good explanation ! thks a lot !

[@АнтонНазарук-щ5с]

Thank you, John! Realy nice tutorial, helped alot.

[@dobcs3236]

Thank you very much 🥰

[@coala6019]

@John Watson Rooney Good tutorial! Thanks! But the lines 23 to 26 are synchrone no?

[@DerekMurawsky]

Subscribed. What are your thoughts on going about it this way vs something like scrapy?

[@JohnWatsonRooney]

Honestly I use scrapy almost all the time now, unless it’s more of a throwaway script

[@phudinhtruong]

great video

[@serge2033]

cool. thanks!!

[@DarkPsychologyIQ]

Hi John, great videos by the way! I was wondering how can I scrape a website for the ASIN's, product title, stock levels and price?

[@GelsYT]

So basically what's happening here in the whole program is that when on the event loop -- all of the tasks while the requests is being made, while it's waitiing it passes the resources to the other task functions? so on and so fort up until we got the response? I'm sorry if it's not that clear

[@JohnWatsonRooney]

Yeah so basically instead of the whole program waiting for a server response it gets on doing other things, ie more requests out. It means there is minimal time spent waiting and maximum time spend doing

[@GelsYT]

@@JohnWatsonRooney THANK YOU! To anyone who watches his videos and does not subscribe! PLEASE DO IT NOW! Thank you man! also please smile more! I like your beard! wish I have that as well

[@GelsYT]

HEEY MAAAN RECENTLY WATCH YOUR VIDS ON SCRAPY! YOU'RE SAVING MY LIFE AGAIN! THANKS1

[@Lorant1984]

Where is the link for the video with "session objects", please?

[@nachoeigu]

How could we use proxy dictionary with aiohttp?

[@peterpann7778]

Hi John, thanks alot for your wonderful vedios, I was wondering which is faster async or multithreading in webscraping?

[@JohnWatsonRooney]

Thanks! Async is faster

[@rangabharath4253]

Awesome 👍😎

[@JohnWatsonRooney]

Thanks!

[@JesusTorres-bt2eb]

Thank you so much, I had learn a lot from your videos, I have a question, It is possible that there is a similar option for pages in cloudflare, I currently use cloudscraper, but it has bugs, do you recommend something?

[@bighneswar98]

The webpage that I'm trying to scrape using asyncio and aiohttp throws an error saying "return self._body.decode( # type: ignore[no-any-return,union-attr] UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 25051-25052: invalid continuation byte" Was working fine in requests and html. What's the solution?

[@nachoeigu]

If you shoud define the best module for web scraping in terms of efficience and robust, what would be? I know selenium, requests, HTMLSessions, aiohttp, AsyncHTMLSession, scrapy, among others. What do you recommend to focus in specifically for its completeness. Thank you for your content.

[@JohnWatsonRooney]

Hi, requests-html is my go to for most smaller projects. Otherwise scrapy

[@anto3617]

Can you explain please how to scrap products price from webstore and send telegram alert when price drop? Thanks for you're video

[@nachoeigu]

Wow, it is an amazing content. Thank you. I would like to know more about asyncio in Python. How did you learn it? Do you recommend some lecture or article?

[@Suresh_Solution]

Bro how to scrape sciencedirect website

[@7488nishant]

Your videos are very informative... Bro....can you make video on web scraping where cookies expires after 30 mins...example website like NSE etc

[@rotatingmind]

Cool tutorial. Just one question: what could we do, if we want to add new urls to the task list from the parsed results?

[@KirillBezzubkine]

I wouldbreally appreciate an answer: i have to make about 10000 api requests but the api supports only about 50 calls at a time so i need to split the whole 10k range into chunks. How do i make async requests in batches.?

[@mohfatkurrozi4069]

Awesome... :-)

[@alexsorrow6133]

Where I can copy this code 🤷🏻‍♂️

[@sadam32]

Hi John, great tutorial again. What theme for VS Code is this? Best regards

[@JohnWatsonRooney]

thanks- i believe that is GruvBox Material. i change a lot!

[@hamidrezarahimi6651]

Link to the quickstart page : docs.aiohttp.org/en/stable/client_quickstart.html

[@powergaming-tu6wj]

ok so i have a question for you i have over 6000+ links and all i want to do is read the url get a one thing off the page and add it to the corelating link in google sheets any tips

[@powergaming-tu6wj]

aka most of the items from the market on a game called runescape

[@terrascape]

hi john , it would be great if you could reply, how do i go about returning the url for each iteration of the event loop?

[@mlgen7]

Thanks for these videos. I am starting my journey on web scraping because I want to try understanding my spend on Deliveroo. (a food delivery app - like uber eats). Some questions and cheekily a tutorial, if possible, would be to learn how to manage social login (I used to google login for Deliveroo) and then be able to go down a level to a page and gather the order history.

[@akalamian]

Nice one, the third lesson learned here! Can you show some database lessons like we scraped 2 sites of shopify which have the same products, then do some price matching work to find the lowest price, thanks

[@JohnWatsonRooney]

Good stuff! Yes I am planning on some more database related things to come

[@akalamian]

@@JohnWatsonRooney absolutely great guide, just scrape 36000 product using 300s, just like a movie:)

[@artabra1019]

thanks .

[@kimkelleher6255]

Hey John!

[@kimkelleher6255]

Also, my next attempt will be grabbing one image from each of the 120k products. Is there a video for that? :)

[@spicer41282]

Nice! More tools in the chest! Thanks John. Any chance you can demo using an if-then-else scraping example? For example: If a set price or a range in price is found. Then scrape that object. Else, move on to the next page? Thanks for considering this request. Or, if you've done something similar in your previous Vids? Can you mention which ones? I've might've missed it... Thanks again!

[@JohnWatsonRooney]

Sure sounds like a good idea I’ll have a look and see what I can come up with!

[@KhalilYasser]

Thank you very much. I got this error `Traceback (most recent call last): File "aiohttp.py", line 34, in results = asyncio.run(main(urls)) File "C:\Users\Future\AppData\Local\Programs\Python\Python37\lib\asyncio unners.py", line 43, in run return loop.run_until_complete(main) File "C:\Users\Future\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 584, in run_until_complete return future.result() File "aiohttp.py", line 18, in main async with aiohttp.ClientSession() as session: AttributeError: module 'aiohttp' has no attribute 'ClientSession'`. Any ideas?

[@KhalilYasser]

The problem occurred because the file name was aiohttp.py (It seems that causes conflict with the aiohttp module). I have put the solution so as to be a reference for others that may this problem happen with.

[@JohnWatsonRooney]

Great glad you worked it out!

[@yuvalbra]

can you publish the source code?

[@mohfatkurrozi4069]

How to fix error 403 Forbidden sir?

[@JohnWatsonRooney]

Try different headers - user agents and cookies to see if that works

[@mohfatkurrozi4069]

@@JohnWatsonRooney yes sir, but i very confuse to fix it, can you some day or maybe later in the next videos. Made a easy to solve my problem forbidden 403.. Before for that.. Thank you sir.. Your replay my comment

[@karthik-ex4dm]

One question...Since this is async, Will it wait for javascript to load fully before fetching the data from a page? Ex: If I put a brand filter in a ecommerce site which updates the page without loading it again, Will Aiohttp be able to fetch the updated the data ??

[@manikandanmanickam9433]

Can you show how to scrap form the delta airlines delta.com

[@JohnWatsonRooney]

Sure I can have a look

[@manikandanmanickam9433]

@@JohnWatsonRooney thanks a lot John

[@manikandanmanickam9433]

Hai John, can you do example for scrap the Google search results with 15 results.(heading, url, content)

[@artabra1019]

grequests is not working ?? i try treturns me none.

[@artabra1019]

wers the source code

[@thekarthik]

You're the best

[@JohnWatsonRooney]

Thank you!

[@MEZHGANO]

This library is great. But how can we pass rich cookies with "path, expires, secure... etc", not just simple "key:value"? I found that documentation is extremely poor in terms of cookies.

[@celerystalk390]

Your videos and topics just keep getting better. Great job!

[@jithin.johnson]

Great tutorial! How do we get around with the IP bans? Bombing the sever with async requests often gets me banned.

[@JohnWatsonRooney]

Thanks! Yes you will need to use some good proxies to avoid the bans. Or I find async is great for getting data from multiple different sites at the same time

[@OprichnikStyle]

why did you put the await in return await r.text? I had to take it out in roder for my code to work

[@justindi6061]

I've learned a lot about scraping through your videos! What is the chances of getting blocked if I host my script on a server and scrape a website multiple times a day? Should I be implementing proxy rotation or can I get away without it?

[@JohnWatsonRooney]

Thanks Justin! It depends on how often is multiple times and how many requests you are making. I’d say try it first and see if you start to run into issues

[@Rani-wm1qq]

Ive been trying to use Selenium to Parse the Name and Price of some coins in a crypto exchange i use very often. But i keep getting this error. driver.find_element_by_xpath("//*[@id="root"]/div/div/div/div[3]/div[2]/div/form/input").click() What do i do

[@Антмара]

Hi John. Thank you for your video. Just one question: Is it possible to make def Pars to be asynсhronous too? I need two functions: first gets links for product card pages from 30 pages of site, and second function uses that links to get products price and other data from every card page. I have made first function asynсhronous and it works really quickly. I want second function o be asynсhronous too because the number of cards is more than 1000. But I still not know, how to gather these two functions together, how to create tasks for them both. Could you please give me your advise how to make it works

[@nahomtsegaye]

you are a wizard my friend

[@tnssajivasudevan1601]

Great tutorial..Sir

[@galichandreyschool]

Very good! :) Thank you!

[@JohnWatsonRooney]

Thanks glad you enjoyed it

[@DM-py7pj]

Will you do a trio video and cover nurseries please?

[@mayurahir9340]

Hello i am doing sum website scraping with one websit but i gating the one problem that website have an button function but when i click it it will give me a one name but wan i cloae the new tabe and reopen the page the name will button givea is hang every time than i can i scrape that Advance thanks and please help me 🙏