Python GC Settings - Change This and Go 20% Faster!

[@daincrawford8439]

Just ran this on one of my scripts with the three lines added. 5 min .3 sec before and 4 min 17.9 sec after. That is amazing.

[@mikeckennedy]

Dain, that is awesome. Thanks for sharing your findings.

[@mrmagnetic927]

Always a big fan of Mike and these amazing videos. I learn so much in this space

[@mikeckennedy]

Thanks so much :)

[@jeanchristophe15]

Awesome video Michael! Thank you!

[@r-tierney]

Nice one! I'll definitely be trying this out. One small thing I'd probably recommend using time.monotonic() to avoid time drift due to NTP / daylight savings etc. when measuring this kind of stuff.

[@MakeDataUseful]

Thanks for the share Michael, I have some very long running data pipelines that I think could benefit from this.

[@mikeckennedy]

You're welcome! Let us know how it turns out.

[@elev007]

Thanks Michael!

[@christophertyler1882]

I have to give this a try. A lot of my data does come from a database but I tend to put into a Pandas dataframe.

[@mikeckennedy]

Excellent. Please report back on whether it makes a difference for you (positive or negative). It's one of those your milage may vary things. With Pandas, the data is in C and outside the GC's purview so it might matter less.

[@blazkowicz666]

Is this valid for python 3.11+?

[@topcivilian]

Why aren't the memory settings optimized by default?

[@qozia1370]

thank you

[@jamesfitzpatrick9607]

A question I need to ask is why are those numbers so low?

[@mikeckennedy]

What numbers are you talking about specifically? And what are you comparing them to?

[@jamesfitzpatrick9607]

@@mikeckennedy kinda all the numbers but mainly the 700. With thinking about this a bit sense I guess this could be a bit of a historical number. On older machines with lower memory, higher numbers could cause problems. So using a 50,000 could be costly. I am now wondering if those numbers will be changed by default in the future as python is trying to make performance improvements.

[@mikeckennedy]

@@jamesfitzpatrick9607 I see. It's interesting that it's 700. That seems insanely low to me. I bet you're right that in 1991 that made sense. It doesn't now. Plus, the thing to keep in mind is that it *only* applies to cycles that are missed by ref counting. Ref counting runs first, then if there are cycles that would have leaked, then the GC kicks in. For 95% of apps, that almost never happens. I'm glad they make it configurable.