How to Test Asynchronous Code in Python

[@ArjanCodes]

👷 Join the FREE Code Diagnosis Workshop to help you review code more effectively using my 3-Factor Diagnosis Framework: www.arjancodes.com/diagnosis.

[@ozoniuss1463]

I love how you explained that asynchronous code helps allocating resources more effectively. Such an elegant way to put it

[@ArjanCodes]

Thank you :) I appreciate the kind words

[@tangentline_]

Could you create a video on how to implement microservices or gRPC with Python please? Great video as always ❤

[@rweaver6]

Bug. Function read_data() in the fetch_event module CANNOT work properly as a coroutine. It will run, but not concurrently. That's because it uses the default Python sleep() function, which *blocks* all loop processing for the duration of the sleep interval. Nothing in the current loop's thread will interrupt the regular Python sleep()'s blocking interval. It's a lengthy blocking call not compatible with concurrent processing of multiple async tasks. Given that sleep() blocks, it is not clear how proper concurrent operation was tested. By contrast, await asyncio.sleep(wait_time) is an async-compatible coroutine and will enable event loop processing during the entire sleep interval, so any other pending tasks, futures or coroutines will run concurrently with that interval, assuming that the loop has already queued those other awaitables for scheduling. Asyncio is *cooperative multitasking*, NOT threading. Threading is *preemptive multitasking*. Threading overhead and difficulties with multi-threaded data access, are what asyncio is designed to avoid. You do NOT want to go multi-threaded without locks on shared data and lots of time budgeted for debugging very difficult issues with shared state in preemptive multitasking. See asyncio documentation in the CPython repository. The Python GIL is another layer of complexity on top of that. I don't claim to understand the GIL.

[@ArjanCodes]

Good catch! We've updated the code example in Git so it now uses the asyncio-compatible sleep call.

[@rweaver6]

@@ArjanCodes Don't forget that as soon as the file opening and reading takes any time at all, it will make evident that open and json.load are also blocking calls and will cause a hiccup in delay. Multiply that by 1000 actual network connections in a production system, and you have *THE* main use case that asyncio is intended to support. The operating system can multitask many parallel sockets. To keep up, an asyncio application has to use asynchronous I/O calls provided by asyncio/aiofile/aoihttp. In summary, all potentially long blocking steps in a coroutine must use asyncio/aio module calls with "await" in front, not generic blocking OS calls. That's cooperative multitasking. (I don't like "concurrent", it's just fancy non-specific jargon). The way this relates to your asyncio testing focus, is that if there is a bug to be found, it could be exactly the use of a non-aio blocking call within a coroutine. Testing for that type of error requires detailed time logging or profiling, verifying overlapped timing. If the time tags of beginning and end of lengthy steps are sequential instead of overlapping, that signifies that concurrency is not being achieved, although all asserts pass, no data are corrupted and no exceptions are thrown. Begging your pardon, I'll stop preaching now, but I'm happy to discuss further, if interested.

[@tajomnynominant]

Just yesterday I was struggling with async unit tests, today I opened YT and saw my favorite YT channel explaining this topic, perfect, thank you! Also great relief when you presented the plugin way, event loop does not look very nice to me... I am trying to avoid complex structures as much as I can.

[@ArjanCodes]

My psychic powers never fail me!

[@TheMgaertne]

pytest-asyncio 0.23.6 just released today supporting latest pytest version.

[@ArjanCodes]

Perfect timing 😎

[@timbrap4693]

Hooray!

[@myworldletsplay]

question open(path) as file is not sync code? if so it is a blocking code in async task, i understand it just some code to show, I just wanna be sure maybe i missed something

[@konstantinmykhailov6708]

BUG in pytest-asyncio I want to point out a recent bug in pytest-asyncio 0.23.x. This version introduced fixture level-dependent event loops (meaning that the event loop for “module” may be different from the event loop for “function”). This may cause issues, especially if the fixtures rely on each other. Here is the specific situation I have encountered: I wanted to establish a client-server connection and use this client as a “session” fixture, and then initialize temporary subscriptions via client on the “function” level (the latter fixture would yield the subscription object and then unsubscribe on teardown). This does not work, since the client created and yielded from the “session” fixture effectively runs in a different loop that the subscription coroutine called in the “function” fixture. I hope it makes sense. There are similar issues described on the pytest-asyncio issue tracker on git. Overall this seems to affect quite a few people, and the only solution for now is to roll back to 0.21. P.s. thanks for the video, Arian! Great, as always.

[@sampri22]

thanks Konstantin for this. it seems in ver pytest-asyncio 0.21 there are no issues with the way you wanted to use session vs. function level fixtures?

[@seselis1]

Thank you! Could you make a tutorial about differences and interactions of multiprocessing+threading+asyncio? There are many classes like ThreadPoolExecutor, ThreadPool, Process etc and it would be great to know when to use what. Some sample code on things like sharing a global resource would also be awesome to see!

[@kellymoses8566]

I like to think of asyncio as parallelized waiting.

[@farazg]

This was timely! Was just about to have to do something like this. Thanks!

[@ArjanCodes]

My psychic powers strike once more!

[@troncooo409]

Hoe does it handle if one of the requests fails or a really long time take to respond?

[@timelschner8451]

Thanks Arjan!

[@ArjanCodes]

You’re welcome!

[@meryplays8952]

Thank you.

[@adigachala-z8o]

Arjan you like you wrote those tests before filming 😂

[@thghtfl]

I wish it wasn’t another toy example viability of which wasn’t even demoed

[@neckbro]

I should share this video with everyone, kids and adults who dream earning 100k as software developers, so they stop dreaming and get a real job instead 😂

[@KD0MOO]

😂