Watch out for this (async) generator cleanup pitfall in Python

[@amarasa2567]

So basically, don't open and close resources in your generators, but give handles to those resources to your generators? Like don't let your generator open a file itself, but give it a file handle?

[@PeterZaitcev]

Exactly.

[@PeterZaitcev]

what *could* be done with the mentioned PEP is to add support for these iterclose methods but do not activate them in any default classes, instead provide a wrapper in contextlib that you can attach to your generator functions (or classes) which adds the missing method.

[@marckiezeender]

How about instead of "don't use generators" we just say "don't use 'with' or 'finally' blocks inside of generators" Edit: what I mean is that you should avoid context and resource management inside of generators. If the generator needs access to, say, a file, then the generator's caller should be responsible for opening and closing that file.

[@AbstractObserver]

He explain why thats not a good way to do that wat in the video

[@themartdog]

​@@AbstractObserver at what timestamp? I didn't hear him say that. Even if he did, I think it's a good idea to avoid with/finally in these contexts

[@aaliboyev]

As a conclusion "Wrap all your with statements with for loop inside into separate function which only holds this statement to guarantee gc"

[@xelaxander]

The resource existing at least as long as the generator seems like to me like the natural solution. Imagine if it wasn’t, I.e. the generator was accessible while the resource isn’t. Suddenly it’s even more cursed, since you can try to get an element from the generator, but it indeterminately fails to yield.

[@xelaxander]

Resources not being garbage-collected immediately is a choice necessary to allow alternative Python implementations with different optimization goals. That said, at least in CPython, you can also just del g, and then gc.collect(). Although I’m not sure if the Python spec guarantees cleanup.

[@DrGreenGiant]

​@@xelaxanderyeah my question would be, is it in the spec or is it an implementation detail. If the latter then I'd happily be that guy who says it sounds like a non cpython implementers problem.

[@aaronm6675]

Wild

[@endoflevelboss]

James Murphy for new BDFL 🎉

[@RogerValor]

Not sure how often I care about when resources are cleaned up, as long as they are cleaned up. But I guess this is good to know for the case when I encounter it. So thx!

[@motbus3]

You can also use a context manager which encloses both generator and resources

[@motbus3]

But that's annoying

[@janhwillems10000]

Thanks a lot James!

[@idoben-yair429]

The ACTUAL solution is to write your code such that it doesn't rely on exactly when resources get cleaned up, and if the exact time of the cleanup is important, then express it EXPLICITLY in the code by calling a cleanup method. RAII-style context managers are not the right tool when the order and timing of cleanup is important. Generators are not the issue here. The engineering approach is the issue. To elaborate: RAII means Resource Acquisition Is Initialization. The converse, i.e., Reference Loss Is Resource Disposal, does NOT hold. This is an important implication of the reference counting style of resource management, unlike C++-style RAII where you can engineer your classes to enforce both RAII and RLIRD.

[@ilmuoui]

Sounds like Rust with extra added steps

[@idoben-yair429]

@@ilmuouievery language has different tools, stop worshipping languages, they’re just that - tools

[@themartdog]

​@@ilmuoui rust itself is programming with extra stepa

[@danwellington3571]

Obligatory Rust comment that completely ignores the 30-odd years of design choices Python has to do its best to not break

[@DrGreenGiant]

Works fine in assembly too

[@tc14hd23]

No problem was ever solved by adding 'async' in front of it

[@fyaa23]

Why do you expect garbage collection to behave like RAII in other languages? The point is that you don't have to take care to clean it up, but you can hardly control when it happens.

[@tophat593]

Sorry, beyond tired and ever so slightly drunk. Want to listen but need to sleep and tomorrow I won't remember this video exists.

[@exploited410]

Answering this so you have a notification tomorrow and will see this video on a clear mind

[@MarianoBustos-i1f]

wake up and watch the video.

[@MrAlanCristhian]

That is the type of behavior that is detected with unit testing.

[@austinnar4494]

That's not true, the async case is literally non-deterministic. So you could have application code that expects the resource to be cleaned up, the unit tests pass, but then fails in production.

[@MrAlanCristhian]

@@austinnar4494 No, in that case the test will randomly fail. Because the test suit it's supose to run multiple times before deployment. The test suit runs every time a change is made.

[@themartdog]

​@@MrAlanCristhian but it could randomly fail only .01% of the time...

[@SpeedingFlare]

Just use C and do if(error) goto cleanup; where cleanup cleans up anything that's non-null in whatever order you want. For async, I don't know