Web Scraping with AIOHTTP and Python

AIOHttp is a client and server side library for Python 3.6 and above that enables us to create http requests asynchronously. It’s fully featured allowing sessions, cookies, custom headers, and everything else you’d expect to see - so naturally I thought it would be a useful tool to share for creating more advanced web scrapers.
When we are scraping data from the web the chances are we will need to make multiple requests to the server to extract the information we are after, given that each of these requests takes time we find that our code is effectively sat waiting for the response from the server before making the next. This slows the process right down. In its simplest form AIOHTTP allows us to use the Python asyncio library to send vast numbers of requests in a short amount of time, letting us create faster and more efficient web scrapers.
Support Me:
Patreon: / johnwatsonrooney (NEW)
Gear Used: jhnwr.com/gear/ (NEW)
Amazon US: amzn.to/2OzqL1M
Amazon UK: amzn.to/2OYuMwo
Hosting: Digital Ocean: m.do.co/c/c7c90f161ff6
-------------------------------------
Disclaimer: These are affiliate links and as an Amazon Associate I earn from qualifying purchases
-------------------------------------
#Timestamps
00:00 Intro
01:17 Docs
02:12 Demo Code
03:54 Web Scraper
09:38 HTML from each page
10:00 Parse HTML
12:10 Expanding Discussion
13:21 Outro

[@Rasstag]

John, Great tutorial... many thanks... now I know how to juggle... ;) Wanted to pass on an observation... apparently, Windows, can be cranky with asyncio/aiohttp. Your example program throws a “RuntimeError: Event loop is closed ” error. However, adding towards the bottom: asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy()) … on top of pages = asyncio.run(main(urls)) … solves whatever 'Event loop' issues that were present.

[@rtxmax8223]

yes i was working on the async request and I got the similar error, the only way it works for me is if my program keeps on running. Since I am doing a constant request response after one ends, i dont get this error. but when I run my program for only one page to scrap then I get this error.

[@luxeave]

@Rasstag, dude ur a life saver. well done!

[@Valentin439]

Thank man!

[@efferington]

Hi John, thank you very much for this. Found this video while trying to figure out how to include an async AIOHTTP loop in some API processing script I'm writing and this was invaluable for figuring out how to structure the code.

[@celerystalk390]

Your videos and topics just keep getting better. Great job!

[@symbolminded5167]

Your video helped a lot, thank you

[@rrahll]

thanks for the tutorial!

[@yacinehechmi6012]

Thank you sir!!

[@GelsYT]

HEEY MAAAN RECENTLY WATCH YOUR VIDS ON SCRAPY! YOU'RE SAVING MY LIFE AGAIN! THANKS1

[@a_private_handle]

Great wallpaper from Firewatch

[@tnssajivasudevan1601]

Great tutorial..Sir

[@andres777video]

No requests from me... just love your videos John! - Thanks for spending the time... I need this code to pull data for 188,600 items (each one is a web page...with 3 tables each) - UIPath would take about 32 days to complete. - asyncio + aiohttp should be much, much faster! Thanks for the tip Rasstag ( had the same issue in Windows)

[@JohnWatsonRooney]

thank you very much!

[@dobcs3236]

Thank you very much 🥰

[@nahomtsegaye]

you are a wizard my friend

[@phudinhtruong]

great video

[@galichandreyschool]

Very good! :) Thank you!

[@JohnWatsonRooney]

Thanks glad you enjoyed it

[@serge2033]

cool. thanks!!

[@rotatingmind]

Cool tutorial. Just one question: what could we do, if we want to add new urls to the task list from the parsed results?

[@fuad471]

Thank you it was really helpful to grasp the asyncio concept

[@coala6019]

@John Watson Rooney Good tutorial! Thanks! But the lines 23 to 26 are synchrone no?

[@adelinabun]

Hi John, great videos by the way! I was wondering how can I scrape a website for the ASIN's, product title, stock levels and price?

[@JesusTorres-bt2eb]

Thank you so much, I had learn a lot from your videos, I have a question, It is possible that there is a similar option for pages in cloudflare, I currently use cloudscraper, but it has bugs, do you recommend something?

[@nachoeigu]

Wow, it is an amazing content. Thank you. I would like to know more about asyncio in Python. How did you learn it? Do you recommend some lecture or article?

[@user-ml5ko3fv8m]

Thank you, John! Realy nice tutorial, helped alot.

[@nsnilesh604]

Awesome way to get list of urls 👌😎

[@JohnWatsonRooney]

Thanks!

[@OprichnikStyle]

why did you put the await in return await r.text? I had to take it out in roder for my code to work

[@mohfatkurrozi4069]

Awesome... :-)

[@tubelessHuma]

Always trying new ways of scraping. Great 👍🌹

[@JohnWatsonRooney]

Thanks!

[@terrascape]

hi john , it would be great if you could reply, how do i go about returning the url for each iteration of the event loop?

[@rangabharath4253]

Awesome 👍😎

[@JohnWatsonRooney]

Thanks!

[@thekarthik]

You're the best

[@JohnWatsonRooney]

Thank you!

[@user-fk9rg8oo4u]

Hi John. Thank you for your video. Just one question: Is it possible to make def Pars to be asynсhronous too? I need two functions: first gets links for product card pages from 30 pages of site, and second function uses that links to get products price and other data from every card page. I have made first function asynсhronous and it works really quickly. I want second function o be asynсhronous too because the number of cards is more than 1000. But I still not know, how to gather these two functions together, how to create tasks for them both. Could you please give me your advise how to make it works

[@user-cr9mf8hp3h]

Really good explanation ! thks a lot !

[@Lorant1984]

Where is the link for the video with "session objects", please?

[@peterpann7778]

Hi John, thanks alot for your wonderful vedios, I was wondering which is faster async or multithreading in webscraping?

[@JohnWatsonRooney]

Thanks! Async is faster

[@nachoeigu]

How could we use proxy dictionary with aiohttp?

[@nachoeigu]

If you shoud define the best module for web scraping in terms of efficience and robust, what would be? I know selenium, requests, HTMLSessions, aiohttp, AsyncHTMLSession, scrapy, among others. What do you recommend to focus in specifically for its completeness. Thank you for your content.

[@JohnWatsonRooney]

Hi, requests-html is my go to for most smaller projects. Otherwise scrapy

[@DerekMurawsky]

Subscribed. What are your thoughts on going about it this way vs something like scrapy?

[@JohnWatsonRooney]

Honestly I use scrapy almost all the time now, unless it’s more of a throwaway script

[@justindi6061]

I've learned a lot about scraping through your videos! What is the chances of getting blocked if I host my script on a server and scrape a website multiple times a day? Should I be implementing proxy rotation or can I get away without it?

[@JohnWatsonRooney]

Thanks Justin! It depends on how often is multiple times and how many requests you are making. I’d say try it first and see if you start to run into issues

[@DM-py7pj]

Will you do a trio video and cover nurseries please?

[@sadam32]

Hi John, great tutorial again. What theme for VS Code is this? Best regards

[@JohnWatsonRooney]

thanks- i believe that is GruvBox Material. i change a lot!

[@kimkelleher6255]

Hey John!

[@kimkelleher6255]

Also, my next attempt will be grabbing one image from each of the 120k products. Is there a video for that? :)

[@anto3617]

Can you explain please how to scrap products price from webstore and send telegram alert when price drop? Thanks for you're video

[@jithin.johnson]

Great tutorial! How do we get around with the IP bans? Bombing the sever with async requests often gets me banned.

[@JohnWatsonRooney]

Thanks! Yes you will need to use some good proxies to avoid the bans. Or I find async is great for getting data from multiple different sites at the same time

[@KirillBezzubkine]

I wouldbreally appreciate an answer: i have to make about 10000 api requests but the api supports only about 50 calls at a time so i need to split the whole 10k range into chunks. How do i make async requests in batches.?

[@spicer41282]

Nice! More tools in the chest! Thanks John. Any chance you can demo using an if-then-else scraping example? For example: If a set price or a range in price is found. Then scrape that object. Else, move on to the next page? Thanks for considering this request. Or, if you've done something similar in your previous Vids? Can you mention which ones? I've might've missed it... Thanks again!

[@JohnWatsonRooney]

Sure sounds like a good idea I’ll have a look and see what I can come up with!

[@bighneswar98]

The webpage that I'm trying to scrape using asyncio and aiohttp throws an error saying "return self._body.decode( # type: ignore[no-any-return,union-attr] UnicodeDecodeError: 'utf-8' codec can't decode bytes in position 25051-25052: invalid continuation byte" Was working fine in requests and html. What's the solution?

[@Suresh_Solution]

Bro how to scrape sciencedirect website

[@GelsYT]

So basically what's happening here in the whole program is that when on the event loop -- all of the tasks while the requests is being made, while it's waitiing it passes the resources to the other task functions? so on and so fort up until we got the response? I'm sorry if it's not that clear

[@JohnWatsonRooney]

Yeah so basically instead of the whole program waiting for a server response it gets on doing other things, ie more requests out. It means there is minimal time spent waiting and maximum time spend doing

[@GelsYT]

@@JohnWatsonRooney THANK YOU! To anyone who watches his videos and does not subscribe! PLEASE DO IT NOW! Thank you man! also please smile more! I like your beard! wish I have that as well

[@powergaming-tu6wj]

ok so i have a question for you i have over 6000+ links and all i want to do is read the url get a one thing off the page and add it to the corelating link in google sheets any tips

[@powergaming-tu6wj]

aka most of the items from the market on a game called runescape

[@alexsorrow6133]

Where I can copy this code 🤷🏻‍♂️

[@Rani-wm1qq]

Ive been trying to use Selenium to Parse the Name and Price of some coins in a crypto exchange i use very often. But i keep getting this error. driver.find_element_by_xpath("//*[@id="root"]/div/div/div/div[3]/div[2]/div/form/input").click() What do i do

[@artabra1019]

thanks .

[@mlgen7]

Thanks for these videos. I am starting my journey on web scraping because I want to try understanding my spend on Deliveroo. (a food delivery app - like uber eats). Some questions and cheekily a tutorial, if possible, would be to learn how to manage social login (I used to google login for Deliveroo) and then be able to go down a level to a page and gather the order history.

[@akalamian]

Nice one, the third lesson learned here! Can you show some database lessons like we scraped 2 sites of shopify which have the same products, then do some price matching work to find the lowest price, thanks

[@JohnWatsonRooney]

Good stuff! Yes I am planning on some more database related things to come

[@akalamian]

@@JohnWatsonRooney absolutely great guide, just scrape 36000 product using 300s, just like a movie:)

[@hamidrezarahimi6651]

Link to the quickstart page : docs.aiohttp.org/en/stable/client_quickstart.html

[@yuvalbra]

can you publish the source code?

[@MEZHGANO]

This library is great. But how can we pass rich cookies with "path, expires, secure... etc", not just simple "key:value"? I found that documentation is extremely poor in terms of cookies.

[@7488nishant]

Your videos are very informative... Bro....can you make video on web scraping where cookies expires after 30 mins...example website like NSE etc

[@karthik-ex4dm]

One question...Since this is async, Will it wait for javascript to load fully before fetching the data from a page? Ex: If I put a brand filter in a ecommerce site which updates the page without loading it again, Will Aiohttp be able to fetch the updated the data ??

[@KhalilYasser]

Thank you very much. I got this error `Traceback (most recent call last): File "aiohttp.py", line 34, in results = asyncio.run(main(urls)) File "C:\Users\Future\AppData\Local\Programs\Python\Python37\lib\asyncio unners.py", line 43, in run return loop.run_until_complete(main) File "C:\Users\Future\AppData\Local\Programs\Python\Python37\lib\asyncio\base_events.py", line 584, in run_until_complete return future.result() File "aiohttp.py", line 18, in main async with aiohttp.ClientSession() as session: AttributeError: module 'aiohttp' has no attribute 'ClientSession'`. Any ideas?

[@KhalilYasser]

The problem occurred because the file name was aiohttp.py (It seems that causes conflict with the aiohttp module). I have put the solution so as to be a reference for others that may this problem happen with.

[@JohnWatsonRooney]

Great glad you worked it out!

[@manikandanmanickam9433]

Can you show how to scrap form the delta airlines delta.com

[@JohnWatsonRooney]

Sure I can have a look

[@manikandanmanickam9433]

@@JohnWatsonRooney thanks a lot John

[@manikandanmanickam9433]

Hai John, can you do example for scrap the Google search results with 15 results.(heading, url, content)

[@mohfatkurrozi4069]

How to fix error 403 Forbidden sir?

[@JohnWatsonRooney]

Try different headers - user agents and cookies to see if that works

[@mohfatkurrozi4069]

@@JohnWatsonRooney yes sir, but i very confuse to fix it, can you some day or maybe later in the next videos. Made a easy to solve my problem forbidden 403.. Before for that.. Thank you sir.. Your replay my comment

[@artabra1019]

grequests is not working ?? i try treturns me none.

[@mayurahir9340]

Hello i am doing sum website scraping with one websit but i gating the one problem that website have an button function but when i click it it will give me a one name but wan i cloae the new tabe and reopen the page the name will button givea is hang every time than i can i scrape that Advance thanks and please help me 🙏

[@artabra1019]

wers the source code