Three ways to optimize your Pandas data frame's memory footprint

If you work with large data sets in Pandas, you'll quickly discover that they take up lots of memory. How can you stop wasting memory, and reduce the size of your data frame? In this video, I show you three techniques you can apply to nearly any project you work on.
The Jupyter notebook for this video (and all of my videos) is at github.com/reuven/youTube-not....
And don't forget my free, weekly "Better developers" newsletter, with new Python articles every week, at BetterDevelopersWeekly.com/.

[@vhphan19]

Prior to doing df.loc[df['total_amount'] >= 0] the dataframe index is a range index (memory efficient). after the .loc operation the index becomes an array with 6 million + entries. Doing a reset_index(drop=True) should reduce your memory usage back to 180Mb

[@ReuvenLerner]

Of course! I was concentrating on the data, but the index obviously takes up memory, as well, and a RangeIndex is far more compact than one with integers (or other objects). I even tell that to my students, but sometimes when recording live, you forget these obvious things. Thanks so much for the comment and correction! (And for a good exercise question to ask in my classes...)

[@aflous]

That's what I was going to comment about. Other than that, this is a very helpful video. Keep'em coming!

[@OhsoLosoo]

I appreciate that you left in that little bit where the memory jumped up unexpectedly. Thanks to another comment when I faced this issue on one of my tests I was able to easily resolve this issue. Keep up the amazing work!

[@ReuvenLerner]

Glad you liked it - I feel it's important to show that yes, I might know more than the average Python/Pandas user, but I definitely make mistakes, forget things, and just plain ol' get things wrong. I've learned a *ton* from my students over the years, and I try to pass along what I learn to future groups. Glad you're enjoying and learning!

[@Vijay-Yarramsetty]

great content. thanks Reuven

[@ReuvenLerner]

So glad to know you're enjoying!

[@gerardohernandez1317]

Thank you for this! It was my first time working with something like this and it kept giving me a warning. Now I know how to fix it! :)

[@ReuvenLerner]

I'm so happy to hear it helped!

[@philtoa334]

Thank for the vidéo , is it possible to change Datetime to 32 bits ?

[@ReuvenLerner]

What a great question! I was sure that datetime64 is all we've got, and so we're stuck with 64 bits. But I did some digging, and it looks like Apache Arrow supports 32-bit times and dates. PyArrow does as well, so *maybe* it's possible to get this to work behind the scenes using specialized dtypes. But it's not a core part of Pandas.

[@philtoa334]

@@ReuvenLerner Thx.

[@jshiff3938]

Obviously the taxi trips with negative distance were subtracting from your memory usage.

[@ReuvenLerner]

Ha! That made me literally laugh out loud...