👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis
@maleldil19 ай бұрын
You should make a video on API rate limiting, especially in async contexts! Also, if you're going to make performance comparisons, you should use a specialised tool like Hyperfine, which runs your program multiple times and gives you stats like average and standard deviation. It's more robust than printing the elapsed time from within the script and running the command by hand multiple times. It would correctly recognise those longer times as outliers.
@IwoGda9 ай бұрын
@@ankit.chaurasia No it can't if we are talking about API with APIKEY for example. Rate limiting is not that easy.
@IwoGda8 ай бұрын
@@ankit.chaurasia I was just responding to your "Rate limiting can easily be bypassed by using proxy", doesn't matter if it's async or not.
@kosmonautofficial2968 ай бұрын
I agree. I thought that aiolimiter would be enough for me and it was but then I ran a larger job unknowingly and it caused database timeouts from the server I was requesting from. I still need to look into it and to slow down this big job I just used an asynchronous lock so I could run that one job slow. I think part of the issue was although I was using aiolimiter and limiting requests, I think there is an httpx connection pool that can have like 100 requests per connection at once. Need to test more though.
@0bl9 ай бұрын
It's also worth noting that HTTPX has HTTP2 support which could be quite helpful in various scenarios.
@kevon2179 ай бұрын
as someone who is new to web development, what are some of those aforementioned various scenarios?
@0bl9 ай бұрын
HTTP/2 support in HTTPX can be particularly beneficial in scenarios where you need to handle multiple requests concurrently, as it allows for more efficient use of network resources. It's also useful in situations where reduced latency and improved page load speeds are critical, like in high-traffic web applications or services.
@anonapache7 ай бұрын
Also worth mentioning is nyquests, comes with automatic handling of http1.1, http2 and http3 out of the box. They also say it's the fastest implementation compared to the standard libs mentioned here.
@PanduPoluan7 ай бұрын
@@anonapacheI can't find "nyquests" anywhere, not in PyPI, not in GitHub...
@trupalcanada9 ай бұрын
License is an important part, i wish you did a video explaining what license a dev should use if they want their work to be open-source or if they want to commercialize it or be both at the same time.
@fortuneosho81379 ай бұрын
I realized the need for httpx when I needed to make three api call before sending a response to the client (each api cost about a sec) imagine waiting for 4 sec before sending a response using requests(totally unacceptable 😂) async does wonders in that area
@ahobbo64018 ай бұрын
Heck yeah. I needed a video on interacting with APIs. I'm working with API's and figuring out how to store API keys securely and the data received from them is my next step. Thanks for covering more options than requests.
@Deadlious9 ай бұрын
For small simple projects it doesn't really matter, what you use. The problems come when you are stuck in deep corporate code, where you need to take into account legacy code, licenses, rate limits and lots of other stuff. Recently, I stumbled on an interesting case regarding certificates and the inability of requests library to handle those properly. I dealt with this via another library, but it took me quite some time to diagnose that the issue is actually with the library and not with the concrete certificate or server. So, from my stand point of view, it would be interesting to see a video about dealing with rate limits, concurrency and certificates.
@cameronball39989 ай бұрын
thanks!! didn’t know about requests sessions. was already using requests in my CLI tool, but now that i just implemented session usage i got 2x performance speed from it 😬 game changer!
@ArjanCodes9 ай бұрын
Great to hear!
@endogeneticgenetics9 ай бұрын
Nice. I'm actually swapping over to HTTPX for my projects at our company -- first notable work should get a working mock this weekend. Great to see see this. Thanks. (Gotta say: I love when blocking ('normal) & non-blocking (e.g. async) are both first class citizens. Makes growing projects easier, and one doesn't know which quick solutions are going to be grown into somethign more serious.
@andrewmenshicov26969 ай бұрын
That's a nice video for a beginner in web requests. What i kinda expected to see there was concurrency with requests using threads. Just to show that it's still possible, but also mention that it's not always the best approach compared to asyncio 😇
@balduinlandolt25549 ай бұрын
Thanks for another useful video, will have to get my hands dirty with sessions, I think. What drives me absolutely nuts with async in Python is that I cannot reasonably limit the concurrency. I may be spoiled from Scala with ZIO, but what I want to be able to do is say "I have 100'000 requests I need to send, send them async, but never more than 8 in parallel because that's what the server can handle", and I still don't know of a good way to do that with asyncio.
@aflous9 ай бұрын
You can easily achieve this with asyncio.Semaphore
@aryankhullar71012 ай бұрын
These videos are very much to the point, I really dig that. Keep making more!
@ArjanCodes2 ай бұрын
Thank you - will do! 💪🏻
@metinEsturb9 ай бұрын
For anything that is being put into production: performance > simplicity. If you put something into production for the first time, you quickly realize how expensive your computational resources are and also how important performance is for the experience of the user, especially if you have solution that must scale well.
@pratikchakravorty9839 ай бұрын
But on the flip side simplicity ensures your software is easier to maintain and allows new engineers to quickly get up to speed with the codebase. Its a trade-off that you need to decide
@JonitoFischer6 ай бұрын
If you want to still use requests in an asynchronous manner, you can use Gevent with monkey patching the socket library. The advantage of Gevent is that you don't have to use async/await and you can still program in plain python.
@markcampanelli9 ай бұрын
raise_for_status is how I typically handle error responses in requests calls. I would love to see examples, esp. with concurrency, that handle responses not on the happy path.
@toutenunmot9 ай бұрын
Just out of curiosity why not setup a venv with python@3.11 to make the demo work with aiohttp ? Also python@3.12.0rc2 ... video production must take a while as 3.12 was released October 2, 2023 and its now 2023-12-16. Video quality and audio are top notch and the explanations are pleasant to listen to, 🙂cheers.
@RocketLR9 ай бұрын
Watching your videos is like a therapy after i've spent a year butchering my python project. Im constantly fighting the decision "Add new feature" vs "take some time to learn better programming" vs "relax so i dont hit the burned out wall".
@baldpolnareff72249 ай бұрын
I literally just introduced httpx in my team, to show a demo where I send some iot data to an API that then plots the charts on its frontend platform, it’s actually quite handy since we mostly used requests before and now I can do async stuff. And of course with pydantic classes it’s a very nice workflow
@rrluthi19 ай бұрын
I've always used AioHttp myself. I just like how it works, it's fast and the concurrent model is nice.
@Phoenix-zk2oe9 ай бұрын
If you're a fan of requests, just use it with a concurrent thread pool; it has a little bit more overhead than async but is as fast as async without wired async await syntax.
@ivanherreros77738 ай бұрын
That was a very nice Arjancodes-style classic video! Thanks. In any case, may we need a very basic introduction to those magic keywords: "await", "asynch", "gather"? Good Software Engineers don't use words they don't understand...
@richsadowsky85809 ай бұрын
Great video. Whether to optimize for simplicity or performance depends on the specific use case. I tend to favor optimizing for time to market and fighting the urge to do premature optimization. Recently, I have had to use async http requests on some new projects where the ability to do concurrent requests is a core project requirement. I have not tried httpx but I like what I saw in this video.
@hansaschauer159 ай бұрын
I personally prefer httpx, too, but for a different reason: it supports the trio async framework out of the box. IMO it is much easier to use (more structured, better API) than AsycIO from the standard lib, in all but the simplest use cases. Thanks for the video! As always, i learned a lot!
@MagnoliaBeats9 ай бұрын
Great video as always, Arjan! I recently named a CustomGPT after you, but it isn’t quite as helpful, lol
@ArjanCodes9 ай бұрын
I'm honored! :) I also made a GPT, you can try it here: chat.openai.com/g/g-h8RebagkT-arjancodes.
@JohnMatthew19 ай бұрын
as always, simple and informative, thank you! Seems like in your last HTTPX example, the printing takes most of the time, maybe put the time capture of just the fetching, not priting. Either way, love the example, thank you.
@Kknewkles9 ай бұрын
Oh, awesome timing. I was thinking of what to move to from requests(thought of httpx).
@ArjanCodes9 ай бұрын
Glad it was helpful!
@hugorimlinger87769 ай бұрын
nice video mate, I had no idea requests had this multiple session patern. Thanks!
@rantalbott69639 ай бұрын
Thumbs up as always, Arjan. The "AIO" is probably an acronym for "Asynch I/O" or something like that, so the letters would be pronounced individually. And in English, the "ai" diphthong would *usually* be pronounced as a long "a", not a long "i". Except, of course, when it's not: it's English. ;-) Something that caught my attention that's worth clarification: the "gather" function in HTTPx. If you have 5 outstanding requests, and one of them somehow goes awry and doesn't finish, are all the others hung forever? Can you, say, specify a timeout and get a return code that says "Not all are done, so you need to check them individually"?
@fortuneosho81379 ай бұрын
Yes you... you can set a raise on 4xx and 5xx errors
@justwoody65117 ай бұрын
Hey Arjan, thanks for the video. This was very helpful for my current work. I was already wondering if there is a way to keep the current session going on. The second improvement with concurrent requests is interesting, but I already using own dedicated threads for each HTTP request. Otherwise I would wait a very long time to gather all the information of many many devices I'm asking for information :D
@saketkr3 ай бұрын
Great work!
@ArjanCodes3 ай бұрын
Thank you, glad you liked it!
@_DRMR_9 ай бұрын
At the end you mention Licensing, but the issue is typically not about "Commercial" usage but of "Proprietary" usage. For instance the GPL licenses allow Commercial, but not Proprietary use.
@jaewilson079 ай бұрын
i've learned so much from your channel! thank you !
@ArjanCodes9 ай бұрын
Glad to hear that! ☺️
@tajomnynominant9 ай бұрын
Also regarding - performance/simplicity - I would say simplicity - it is simply in Python's DNA, unlike performance :)
@youtubeenjoyer17438 ай бұрын
Popular python libraries suffer bad API design. Check out the two most popular web frameworks: Flask and FastAPI. If you look at them closely, you see absolutely inexcusable design decisions. It seems that most devs see the "look how compact the code is to run this hello world example" in the QuickStart session of the doc and then completely overlook everything else.
@jcurwen317 ай бұрын
4 years ago, i used a library named grequests, for requests + Gevent. It worked great for a project where i a had to fetch about 50 invoices in pdf forlat from an api. Very simple to use. I will give a try to httpx.
@DenizDemirsoy9 ай бұрын
Re. the httpx option: If the API has rate limiting where should a strategy like exponential backoff be implemented? Great video BTW. Thanks.
@Atimska9 ай бұрын
I'm a huge fan of asdf for managing multiple versions of python (and it has many other plugins). Just mentioning it in case you have a package that requires a specific python version (3.11 vs 3.12 for example) you can easily have them both installed and switch around based on different projects.
@revenez9 ай бұрын
Useful and clear video, thank you! 👍
@ArjanCodes9 ай бұрын
You’re welcome!
@Michael2010789 ай бұрын
Great explanation. Many thanks!
@jean-marcfraisse71919 ай бұрын
👍excellent video, as always! Personally, I try to never favour simplicity over performance or the other way around. I try to assess the performance needs for a project, depending on its specifics: most of the time, at this stage, performance does not appear as being a critical issue, but sometimes you can anticipate that it will: I just try not to be over-pessimistic and not give in to premature optimisation. Then I go with the simplest possible implementation (the fastest, easiest to write, easiest to read, and in most cases easiest to maintain or change later, option: in that case, would be requests) in order to get the functionality up and running. Then, if tests show that things go smoothly, there is no need for perfomance improvements. On the other hand, if the test results show "poor" performance, it means that it's time to improve, and possibly to switch libs/modules. "Poor" here isn't something absolute, it really depends on the project and context(s). For some projects, a 10ms improvement makes a huge final difference, for some other ones, not really... So, yes, after writing, I realise that I do tend to favour simplicity over performance hahahaha 😅😂 but I try to do it in a way that is never detrimental to the project. I prefer simplicity (who doesn't?) until this simplicity becomes a bottleneck. In short: use the right tool for the job 😅
@Manitary9 ай бұрын
7:00 aiohttp does work with Python 3.12 starting from version 3.9
@francoisschoeman53509 ай бұрын
This was really cool and really useful. Thank you!
@ArjanCodes9 ай бұрын
Glad it was helpful!
@VolodymyrMoon2 ай бұрын
Love httpx❤ While it still lacks elastic retry policy like requests
@Lexaire9 ай бұрын
Nice to see the options available, but you didn't delve into any feature comparisons for advanced things like streaming post bodies which aiohttp handles better than httpx.
@cetilly9 ай бұрын
aiohttp is the best. Period. HTTPX is flaky and as you saw in this video is unpredictable. I never encounter these issues with aiohttp; it just always works.
@ErikS-9 ай бұрын
"it just always works" - Didn't Arjan show that aiohhtp doesn't (yet) work on python 3.12😉
@cetilly9 ай бұрын
@@ErikS- Touché. But it’s absolutely solid on 3.11
@Casimistico9 ай бұрын
Great video as always.
@ArjanCodes9 ай бұрын
Glad you enjoyed it!
@eesveryowncarrzkiss85965 ай бұрын
I love the keyboard in your into. Could you please provide the make and model number?
@shizueigaki7029 ай бұрын
Would love a deep dive into Async programming with Python for backend web-devs.
@sevbo9 ай бұрын
Happy watching)
@Asand3r9 ай бұрын
Thanks, that's really awesome. =)
@ArjanCodes9 ай бұрын
Thank you!
@tajomnynominant9 ай бұрын
Great comparison, as always, thanks for that sir! I was familiar with requests only by this point. Objection your honor: if you were to use larger sample, or more endpoints, the difference would be more comparable. Anyway, I have a question to our typing guru: is it allright to use Any for requests (or other libs) response? I always struggle with this one, just hate putting Any, but what do I know what would I get in return of the endpoint?
@Proprogrammer0019 ай бұрын
I'm confused, I understand connection pooling, but what exactly is the benefit of using aiohttp or httpx over simply using threads? The code is pretty simple IMO, where you just have futures instead of the async await syntax, AND you don't rely on any external lib.
@ramonbraga10249 ай бұрын
What a uplifting content. I felt compelled to comment so that u you receive feedback on how good your content about Python is. Congrats and happy 2024. Waiting for you to add some more content on the course to buy your python one.
@ArjanCodes9 ай бұрын
I appreciate that! Happy holidays! :)
@conconmc9 ай бұрын
Simplicity until you need performance! (and consider switching to Go or Rust lol)
@GOTHICforLIFE19 ай бұрын
Rust is not much simplicity (Unless you compare it to C++), and imo overkill in almost all scenarios. Go on the other hand is a great middle option - Slightly more tedious to write than Python, but a lot(!) more performance.
@arpitkumar45254 ай бұрын
What if we use it to make a small internal tool at a company? Do we still need license for that?
@ImARichard9 ай бұрын
Were you using a VPN in this video? It appears that your IP may be exposed if not. Not sure if you care about that though, but figured Id share. Awesome video. Ive been stuck using requests for so long that I havent even considered looking at alternatives!
@kapustinalexander8489 ай бұрын
I think for aiohttp + one `with` statement does not add some complexity - this is just another way of code structure. Maybe interesting that httpx works on top of anyio, so it can be runned on both: asyncio and trio
@victorbrylew17759 ай бұрын
requests for simple scripts and pocs. aiohttp for production
@yickysan9 ай бұрын
Seeing Arjan use type Any. My whole world has crumbled.
@omerpriel55889 ай бұрын
Thanks
@quillaja9 ай бұрын
Isn't it pretty simple to add concurrency to requests by just wrapping each request in a future/promise (whatever it is in python), making it essentially identical to aoihttp and httpx?
@brSUCU2 ай бұрын
Great
@ItsStaffDaddy9 ай бұрын
I started with aiohttp but I much prefer httpx
@MicheleHjorleifsson9 ай бұрын
performance vs. simplicity.. depends on the use case. If i just need to get an api call, simplicity, if i am making lots of them and i need the code to be performant... performance
@PanduPoluan7 ай бұрын
It's surprising that aiohttp have problems with 3.12 ... makes me happy that I had settled with httpx for my asynchronous HTTP needs. A friend of mine need to query several million data points to grab JPEG files. Her original non-async code takes _days_ to complete. With concurrency (I suggested to her to use httpx), her script can now finish overnight. Her code has like 600 requests on the fly at any given time, and leveraging HTTP/2 which is supported by httpx. She leverages asyncio.wait() to be able to gather only completed tasks and handle the results as they come in, generating additional tasks periodically to keep the number of in-session requests at 600.
@claudiocl89379 ай бұрын
async for the win
@cesarlinares81499 ай бұрын
We are all using python... Simplicity first
@Doppelwulf8 ай бұрын
I am surprised you aren't using venvs to run multiple Python versions.
@mariuszp16499 ай бұрын
If aiohttp has issues in python 3.12, httpx gets a chance to catch up in popularity.
@WarbossPepe9 ай бұрын
im a noob programmer so im gonna go with simplicity > performance. But i doubt a business application is gonna opt for that
@carloszenteno9 ай бұрын
I wished you had included Axios on this comparison.
@nomonkey4169 ай бұрын
Why main func needs the async keyword?
@thierrylaude60879 ай бұрын
I use requests with gevent for my scrapers and I don't need all this dirty async boilerplate code to get comparable performance as httpx, without any headache. For now, I don't see the interest of httpx and aiohttp.
@MrGodLiike9 ай бұрын
Does it make sense to use connection pooling, when I just send out one request?
@Kknewkles9 ай бұрын
Oh! Help me out here: with's going on with `func(session: requests.Session)`? Doesn't look like either a type hint or a named parameter assignment. I'm a decent Python programmer, but my knowledge of its features is quite limited :^)
@Tudorabil9 ай бұрын
it's a typehint, since it's after a : symbol. it looks different because it was imported from the package requests. One could also import Session directly like from requests import Session and just have that as a type it. It doesn't really matter and it depends on the coding style mostly
@Michallote9 ай бұрын
Just a typehint telling exactly what object it is expecting
@Kknewkles9 ай бұрын
@@Michallote ah, I expected -> for each hint, but it's only used for the return type, right?
@_Akhilleus_9 ай бұрын
@@Kknewkles the -> is for the return type of the function. The : is for the parameter's type
@Kknewkles9 ай бұрын
@@_Akhilleus_ right. I've used it once or twice and forgot that since.
@21Kip6 ай бұрын
what about niquests?
@kursatcalk18506 ай бұрын
aiosonic is now the fastest.
@MrFibbanacci9 ай бұрын
Hmm maybe a video about licenses would be a good idea?
@_Akhilleus_9 ай бұрын
Wasn't requests replaced by urllib in standard python libraries?