Requests vs HTTPX vs Aiohttp

[@ArjanCodes]

👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis

[@maleldil1]

You should make a video on API rate limiting, especially in async contexts! Also, if you're going to make performance comparisons, you should use a specialised tool like Hyperfine, which runs your program multiple times and gives you stats like average and standard deviation. It's more robust than printing the elapsed time from within the script and running the command by hand multiple times. It would correctly recognise those longer times as outliers.

[@IwoGda]

@@ankit.chaurasia No it can't if we are talking about API with APIKEY for example. Rate limiting is not that easy.

[@IwoGda]

@@ankit.chaurasia I was just responding to your "Rate limiting can easily be bypassed by using proxy", doesn't matter if it's async or not.

[@kosmonautofficial296]

I agree. I thought that aiolimiter would be enough for me and it was but then I ran a larger job unknowingly and it caused database timeouts from the server I was requesting from. I still need to look into it and to slow down this big job I just used an asynchronous lock so I could run that one job slow. I think part of the issue was although I was using aiolimiter and limiting requests, I think there is an httpx connection pool that can have like 100 requests per connection at once. Need to test more though.

[@VivekKumarSinha25]

Rate limiting as is limiting the number of incoming requests or outgoing requests. Both have different use cases. In case of later, when we need to make 100s of API calls in order to, say, collect data on regular basis, asyncio has something called semaphore, that is used to limit the number of concurrent requests at any given time. By default, this concurrency limit is set to 100. Another advantage of asyncio is that, if a request doesn't return a response in 5 minutes, it'll automatically cancel the task associated with it. Can save you a lot of time as compared to requests, which, in my experience, can lead to service downtime in case of a multithreaded application. Rate limiting for incoming requests is another topic which I don't have much experience about, so won't comment on that.

[@0bl]

It's also worth noting that HTTPX has HTTP2 support which could be quite helpful in various scenarios.

[@kevon217]

as someone who is new to web development, what are some of those aforementioned various scenarios?

[@0bl]

HTTP/2 support in HTTPX can be particularly beneficial in scenarios where you need to handle multiple requests concurrently, as it allows for more efficient use of network resources. It's also useful in situations where reduced latency and improved page load speeds are critical, like in high-traffic web applications or services.

[@anonapache]

Also worth mentioning is nyquests, comes with automatic handling of http1.1, http2 and http3 out of the box. They also say it's the fastest implementation compared to the standard libs mentioned here.

[@PanduPoluan]

​@@anonapacheI can't find "nyquests" anywhere, not in PyPI, not in GitHub...

[@trupalcanada]

License is an important part, i wish you did a video explaining what license a dev should use if they want their work to be open-source or if they want to commercialize it or be both at the same time.

[@ahobbo6401]

Heck yeah. I needed a video on interacting with APIs. I'm working with API's and figuring out how to store API keys securely and the data received from them is my next step. Thanks for covering more options than requests.

[@einstein_god]

I realized the need for httpx when I needed to make three api call before sending a response to the client (each api cost about a sec) imagine waiting for 4 sec before sending a response using requests(totally unacceptable 😂) async does wonders in that area

[@rahulburman77]

@@danielschmider5069 But in this video, he said async is not available with Requests.

[@Deadlious]

For small simple projects it doesn't really matter, what you use. The problems come when you are stuck in deep corporate code, where you need to take into account legacy code, licenses, rate limits and lots of other stuff. Recently, I stumbled on an interesting case regarding certificates and the inability of requests library to handle those properly. I dealt with this via another library, but it took me quite some time to diagnose that the issue is actually with the library and not with the concrete certificate or server. So, from my stand point of view, it would be interesting to see a video about dealing with rate limits, concurrency and certificates.

[@aryankhullar7101]

These videos are very much to the point, I really dig that. Keep making more!

[@ArjanCodes]

Thank you - will do! 💪🏻

[@Manitary]

7:00 aiohttp does work with Python 3.12 starting from version 3.9

[@cameronball3998]

thanks!! didn’t know about requests sessions. was already using requests in my CLI tool, but now that i just implemented session usage i got 2x performance speed from it 😬 game changer!

[@ArjanCodes]

Great to hear!

[@endogeneticgenetics]

Nice. I'm actually swapping over to HTTPX for my projects at our company -- first notable work should get a working mock this weekend. Great to see see this. Thanks. (Gotta say: I love when blocking ('normal) & non-blocking (e.g. async) are both first class citizens. Makes growing projects easier, and one doesn't know which quick solutions are going to be grown into somethign more serious.

[@markcampanelli]

raise_for_status is how I typically handle error responses in requests calls. I would love to see examples, esp. with concurrency, that handle responses not on the happy path.

[@andrewmenshicov2696]

That's a nice video for a beginner in web requests. What i kinda expected to see there was concurrency with requests using threads. Just to show that it's still possible, but also mention that it's not always the best approach compared to asyncio 😇

[@balduinlandolt2554]

Thanks for another useful video, will have to get my hands dirty with sessions, I think. What drives me absolutely nuts with async in Python is that I cannot reasonably limit the concurrency. I may be spoiled from Scala with ZIO, but what I want to be able to do is say "I have 100'000 requests I need to send, send them async, but never more than 8 in parallel because that's what the server can handle", and I still don't know of a good way to do that with asyncio.

[@aflous]

You can easily achieve this with asyncio.Semaphore

[@baldpolnareff7224]

I literally just introduced httpx in my team, to show a demo where I send some iot data to an API that then plots the charts on its frontend platform, it’s actually quite handy since we mostly used requests before and now I can do async stuff. And of course with pydantic classes it’s a very nice workflow

[@rrluthi1]

I've always used AioHttp myself. I just like how it works, it's fast and the concurrent model is nice.

[@hugorimlinger8776]

nice video mate, I had no idea requests had this multiple session patern. Thanks!

[@Kknewkles]

Oh, awesome timing. I was thinking of what to move to from requests(thought of httpx).

[@ArjanCodes]

Glad it was helpful!

[@richsadowsky8580]

Great video. Whether to optimize for simplicity or performance depends on the specific use case. I tend to favor optimizing for time to market and fighting the urge to do premature optimization. Recently, I have had to use async http requests on some new projects where the ability to do concurrent requests is a core project requirement. I have not tried httpx but I like what I saw in this video.

[@metinEsturb]

For anything that is being put into production: performance > simplicity. If you put something into production for the first time, you quickly realize how expensive your computational resources are and also how important performance is for the experience of the user, especially if you have solution that must scale well.

[@pratikchakravorty983]

But on the flip side simplicity ensures your software is easier to maintain and allows new engineers to quickly get up to speed with the codebase. Its a trade-off that you need to decide

[@tammoton9494]

Very good, to the point video.

[@ArjanCodes]

Glad you liked it!

[@toutenunmot]

Just out of curiosity why not setup a venv with python@3.11 to make the demo work with aiohttp ? Also python@3.12.0rc2 ... video production must take a while as 3.12 was released October 2, 2023 and its now 2023-12-16. Video quality and audio are top notch and the explanations are pleasant to listen to, 🙂cheers.

[@ivanherreros7773]

That was a very nice Arjancodes-style classic video! Thanks. In any case, may we need a very basic introduction to those magic keywords: "await", "asynch", "gather"? Good Software Engineers don't use words they don't understand...

[@JohnMatthew1]

as always, simple and informative, thank you! Seems like in your last HTTPX example, the printing takes most of the time, maybe put the time capture of just the fetching, not priting. Either way, love the example, thank you.

[@jaewilson07]

i've learned so much from your channel! thank you !

[@ArjanCodes]

Glad to hear that! ☺️

[@RocketLR]

Watching your videos is like a therapy after i've spent a year butchering my python project. Im constantly fighting the decision "Add new feature" vs "take some time to learn better programming" vs "relax so i dont hit the burned out wall".

[@CleverBoy0098]

If you're a fan of requests, just use it with a concurrent thread pool; it has a little bit more overhead than async but is as fast as async without wired async await syntax.

[@Michael201078]

Great explanation. Many thanks!

[@MagnoliaBeats]

Great video as always, Arjan! I recently named a CustomGPT after you, but it isn’t quite as helpful, lol

[@ArjanCodes]

I'm honored! :) I also made a GPT, you can try it here: chat.openai.com/g/g-h8RebagkT-arjancodes.

[@jcurwen31]

4 years ago, i used a library named grequests, for requests + Gevent. It worked great for a project where i a had to fetch about 50 invoices in pdf forlat from an api. Very simple to use. I will give a try to httpx.

[@revenez]

Useful and clear video, thank you! 👍

[@ArjanCodes]

You’re welcome!

[@DenizDemirsoy]

Re. the httpx option: If the API has rate limiting where should a strategy like exponential backoff be implemented? Great video BTW. Thanks.

[@hansaschauer15]

I personally prefer httpx, too, but for a different reason: it supports the trio async framework out of the box. IMO it is much easier to use (more structured, better API) than AsycIO from the standard lib, in all but the simplest use cases. Thanks for the video! As always, i learned a lot!

[@rantalbott6963]

Thumbs up as always, Arjan. The "AIO" is probably an acronym for "Asynch I/O" or something like that, so the letters would be pronounced individually. And in English, the "ai" diphthong would *usually* be pronounced as a long "a", not a long "i". Except, of course, when it's not: it's English. ;-) Something that caught my attention that's worth clarification: the "gather" function in HTTPx. If you have 5 outstanding requests, and one of them somehow goes awry and doesn't finish, are all the others hung forever? Can you, say, specify a timeout and get a return code that says "Not all are done, so you need to check them individually"?

[@einstein_god]

Yes you... you can set a raise on 4xx and 5xx errors

[@Atimska]

I'm a huge fan of asdf for managing multiple versions of python (and it has many other plugins). Just mentioning it in case you have a package that requires a specific python version (3.11 vs 3.12 for example) you can easily have them both installed and switch around based on different projects.

[@justwoody6511]

Hey Arjan, thanks for the video. This was very helpful for my current work. I was already wondering if there is a way to keep the current session going on. The second improvement with concurrent requests is interesting, but I already using own dedicated threads for each HTTP request. Otherwise I would wait a very long time to gather all the information of many many devices I'm asking for information :D

[@francoisschoeman5350]

This was really cool and really useful. Thank you!

[@ArjanCodes]

Glad it was helpful!

[@arpitkumar4525]

What if we use it to make a small internal tool at a company? Do we still need license for that?

[@Casimistico]

Great video as always.

[@ArjanCodes]

Glad you enjoyed it!

[@JonitoFischer]

If you want to still use requests in an asynchronous manner, you can use Gevent with monkey patching the socket library. The advantage of Gevent is that you don't have to use async/await and you can still program in plain python.

[@tajomnynominant]

Great comparison, as always, thanks for that sir! I was familiar with requests only by this point. Objection your honor: if you were to use larger sample, or more endpoints, the difference would be more comparable. Anyway, I have a question to our typing guru: is it allright to use Any for requests (or other libs) response? I always struggle with this one, just hate putting Any, but what do I know what would I get in return of the endpoint?

[@Lexaire]

Nice to see the options available, but you didn't delve into any feature comparisons for advanced things like streaming post bodies which aiohttp handles better than httpx.

[@_DRMR_]

At the end you mention Licensing, but the issue is typically not about "Commercial" usage but of "Proprietary" usage. For instance the GPL licenses allow Commercial, but not Proprietary use.

[@jean-marcfraisse7191]

👍excellent video, as always! Personally, I try to never favour simplicity over performance or the other way around. I try to assess the performance needs for a project, depending on its specifics: most of the time, at this stage, performance does not appear as being a critical issue, but sometimes you can anticipate that it will: I just try not to be over-pessimistic and not give in to premature optimisation. Then I go with the simplest possible implementation (the fastest, easiest to write, easiest to read, and in most cases easiest to maintain or change later, option: in that case, would be requests) in order to get the functionality up and running. Then, if tests show that things go smoothly, there is no need for perfomance improvements. On the other hand, if the test results show "poor" performance, it means that it's time to improve, and possibly to switch libs/modules. "Poor" here isn't something absolute, it really depends on the project and context(s). For some projects, a 10ms improvement makes a huge final difference, for some other ones, not really... So, yes, after writing, I realise that I do tend to favour simplicity over performance hahahaha 😅😂 but I try to do it in a way that is never detrimental to the project. I prefer simplicity (who doesn't?) until this simplicity becomes a bottleneck. In short: use the right tool for the job 😅

[@saketkr]

Great work!

[@ArjanCodes]

Thank you, glad you liked it!

[@Proprogrammer001]

I'm confused, I understand connection pooling, but what exactly is the benefit of using aiohttp or httpx over simply using threads? The code is pretty simple IMO, where you just have futures instead of the async await syntax, AND you don't rely on any external lib.

[@eesveryowncarrzkiss8596]

I love the keyboard in your into. Could you please provide the make and model number?

[@Asand3r]

Thanks, that's really awesome. =)

[@ArjanCodes]

Thank you!

[@kapustinalexander848]

I think for aiohttp + one `with` statement does not add some complexity - this is just another way of code structure. Maybe interesting that httpx works on top of anyio, so it can be runned on both: asyncio and trio

[@MicheleHjorleifsson]

performance vs. simplicity.. depends on the use case. If i just need to get an api call, simplicity, if i am making lots of them and i need the code to be performant... performance

[@sevbo]

Happy watching)

[@quillaja]

Isn't it pretty simple to add concurrency to requests by just wrapping each request in a future/promise (whatever it is in python), making it essentially identical to aoihttp and httpx?

[@MrGodLiike]

Does it make sense to use connection pooling, when I just send out one request?

[@VolodymyrMoon]

Love httpx❤ While it still lacks elastic retry policy like requests

[@ramonbraga1024]

What a uplifting content. I felt compelled to comment so that u you receive feedback on how good your content about Python is. Congrats and happy 2024. Waiting for you to add some more content on the course to buy your python one.

[@ArjanCodes]

I appreciate that! Happy holidays! :)

[@shizueigaki702]

Would love a deep dive into Async programming with Python for backend web-devs.

[@Kknewkles]

Oh! Help me out here: with's going on with `func(session: requests.Session)`? Doesn't look like either a type hint or a named parameter assignment. I'm a decent Python programmer, but my knowledge of its features is quite limited :^)

[@Tudorabil]

it's a typehint, since it's after a : symbol. it looks different because it was imported from the package requests. One could also import Session directly like from requests import Session and just have that as a type it. It doesn't really matter and it depends on the coding style mostly

[@Michallote]

Just a typehint telling exactly what object it is expecting

[@Kknewkles]

@@Michallote ah, I expected -> for each hint, but it's only used for the return type, right?

[@_Akhilleus_]

​​​@@Kknewkles the -> is for the return type of the function. The : is for the parameter's type

[@Kknewkles]

@@_Akhilleus_ right. I've used it once or twice and forgot that since.

[@nomonkey416]

Why main func needs the async keyword?

[@tajomnynominant]

Also regarding - performance/simplicity - I would say simplicity - it is simply in Python's DNA, unlike performance :)

[@youtubeenjoyer1743]

Popular python libraries suffer bad API design. Check out the two most popular web frameworks: Flask and FastAPI. If you look at them closely, you see absolutely inexcusable design decisions. It seems that most devs see the "look how compact the code is to run this hello world example" in the QuickStart session of the doc and then completely overlook everything else.

[@cetilly]

aiohttp is the best. Period. HTTPX is flaky and as you saw in this video is unpredictable. I never encounter these issues with aiohttp; it just always works.

[@ErikS-]

"it just always works" - Didn't Arjan show that aiohhtp doesn't (yet) work on python 3.12😉

[@cetilly]

@@ErikS- Touché. But it’s absolutely solid on 3.11

[@ImARichard]

Were you using a VPN in this video? It appears that your IP may be exposed if not. Not sure if you care about that though, but figured Id share. Awesome video. Ive been stuck using requests for so long that I havent even considered looking at alternatives!

[@WarbossPepe]

im a noob programmer so im gonna go with simplicity > performance. But i doubt a business application is gonna opt for that

[@carloszenteno]

I wished you had included Axios on this comparison.

[@omerpriel5588]

Thanks

[@21Kip]

what about niquests?

[@Doppelwulf]

I am surprised you aren't using venvs to run multiple Python versions.

[@victorbrylew1775]

requests for simple scripts and pocs. aiohttp for production

[@thierrylaude6087]

I use requests with gevent for my scrapers and I don't need all this dirty async boilerplate code to get comparable performance as httpx, without any headache. For now, I don't see the interest of httpx and aiohttp.

[@ItsStaffDaddy]

I started with aiohttp but I much prefer httpx

[@claudiocl8937]

async for the win

[@yickysan]

Seeing Arjan use type Any. My whole world has crumbled.

[@conconmc]

Simplicity until you need performance! (and consider switching to Go or Rust lol)

[@GOTHICforLIFE1]

Rust is not much simplicity (Unless you compare it to C++), and imo overkill in almost all scenarios. Go on the other hand is a great middle option - Slightly more tedious to write than Python, but a lot(!) more performance.

[@brSUCU]

Great

[@cesarlinares8149]

We are all using python... Simplicity first

[@mariuszp1649]

If aiohttp has issues in python 3.12, httpx gets a chance to catch up in popularity.

[@MrFibbanacci]

Hmm maybe a video about licenses would be a good idea?

[@_Akhilleus_]

Wasn't requests replaced by urllib in standard python libraries?

[@aflous]

Requests uses it under the hood

[@kursatcalk1850]

aiosonic is now the fastest.

[@PanduPoluan]

It's surprising that aiohttp have problems with 3.12 ... makes me happy that I had settled with httpx for my asynchronous HTTP needs. A friend of mine need to query several million data points to grab JPEG files. Her original non-async code takes _days_ to complete. With concurrency (I suggested to her to use httpx), her script can now finish overnight. Her code has like 600 requests on the fly at any given time, and leveraging HTTP/2 which is supported by httpx. She leverages asyncio.wait() to be able to gather only completed tasks and handle the results as they come in, generating additional tasks periodically to keep the number of in-session requests at 600.