Stop using inplace=True in Pandas!

Рет қаралды 3,447

Python and Pandas with Reuven Lerner

Күн бұрын

Lots of Pandas users think that inplace=True is a great way to save memory, to make their queries more efficient, or generally to be better. But none of these is true. Moreover, inplace=True is going away in the future. So if you're using it, you should stop! In this video, I demonstrate what inplace=True means and does, what you can (should) do instead, and some of the methods that are affected.

Пікірлер: 30

@christophertyler1882 Жыл бұрын

I stopped using inplace=True once I started doing method chaining. The other bonus is that you can play with the data without changing the original dataframe, then once everything is good during EDA, you can then overwrite the original dataframe later.

@ReuvenLerner Жыл бұрын

I totally agree!

@BrownStain_Silver 4 ай бұрын

Awesome advice. I'm working in Pandas to subset database records. I was considering inplace=True for memory efficiency. Ill assign back to the original variable for the reasons mentioned. Thanks for the tip.

@ReuvenLerner 4 ай бұрын

Glad it helped!

@BrownStain_Silver 4 ай бұрын

That's not intuitively what a new Pandas user might expect inplace=True to do. System memory will definitely be a major concern for my work project that others will use on aged government computers. I've prototyped on a small scale but need to scale it up. Definitely helpful advice. Thank you!

@johnbainbridge1931 9 ай бұрын

What about in a loop scenario, eg preparing train and test? data=[train, test] for df in data: df.drop(columns = ['BLOCKID','SUMLEVEL','primary'], inplace=True ) thanks

@ReuvenLerner 9 ай бұрын

The Pandas core developers have said, for several years now, that using inplace=True is deprecated, provides no advantages, and will go away in a future version. For that reason alone, I would avoid using it. The example you gave could be handled with tuple unpacking and a list comprehension: train, test = [df.drop(columns=['BLOCKID', 'SUMLEVEL', 'primary'] for df in [train, test]] There are probably other, better ways to do it, but this strikes me as a pretty good and quick alternative.

@sloanlance Жыл бұрын

I think it's pretty important to lead off with the fact that the Pandas development team is working to remove "inplace" support. Then mention chaining and no performance/memory improvement.

@ReuvenLerner Жыл бұрын

Sorry if I didn't make it clear enough -- inplace=True is a bad idea for a whole lot of reasons, and the core Pandas developers have made it clear that it's going to go away at some point. They've been saying this for a long time, so it's easy to say that we can ignore this statement, but one of these days, people will get an unpleasant surprise. I keep seeing people use inplace=True in my Pandas courses, and they're often convinced that it's the right thing to do... so I decided to give them a bit of a video scolding.

@tushartiwari7929 Жыл бұрын

Can you point where Pandas core developer have mentioned that it does not save memory? I was in illusion that it saves memory.

@ReuvenLerner Жыл бұрын

I've seen it in a bunch of places, but here's a core Pandas developer saying that it'll go away in Pandas 2.0: github.com/pandas-dev/pandas/issues/16529 That clearly didn't happen, but we should expect it to be the case in the near future. Update: I didn't really answer your question, did I? Whoops! More below... I've definitely seen the core developers say that you cannot know what's going on behind the scenes. It feels like inplace=True will save memory, but you cannot be sure of this, in any way. Even if you don't see a new data frame, one is almost certainly being created behind the scenes. Here are some places where it is at least mentioned that you shouldn't use it and/or that there's more memory use than meets the eye: • www.dataschool.io/future-of-pandas/#inplace • github.com/pandas-dev/pandas/issues/48682 So... I didn't see a direct statement from the core developers. But it has been mentioned so often that I'm sure there is a source for this that I just didn't find.

@tushartiwari7929 Жыл бұрын

@@ReuvenLerner Got it. There is intent to remove inplace at various issues on Github. Thanks for the detailed response.

@Arne_Boeses Жыл бұрын

Thanks for the tip! Will implement that from now on 😊

@WildRover1964 Жыл бұрын

Interesting and I'll bear that in mind. But didn't you once say that setting a dataframe back to itself (ie df=df.reset_index(), or somesuch) was problematic? Can't remember why. Don't you have to force it to make a copy or something like that. That's why I've been using Inplace=True up to now. ps I really enjoy these bite-sized insights

@ReuvenLerner Жыл бұрын

I'm glad you're enjoying the videos! As for whether inplace=True is a good idea, I might have said, years ago, that this is the case. But for a while now, it has been known that inplace=True doesn't save any memory, and that the odds are good that it'll be going away in a future version of Pandas.

@elu1 Жыл бұрын

Thanks. Basically it says to apply method chaining on the fly.

@mrityunjaypathak8792 Жыл бұрын

never thought this . Thanks

@franky12 Жыл бұрын

I also thought that i would save memory by using "inplace=True", will stop using it from now on, thanks for the hint.

@ReuvenLerner Жыл бұрын

You're not alone in thinking this! Glad I could help.

@method341 Жыл бұрын

I had the None problem today and could not figure out why. Once I removed inplace=True, it worked.

@ReuvenLerner Жыл бұрын

Excellent!

@lade_edal Жыл бұрын

Not the most convincing video. The memory issue aside the rest of this was just filler of basic play around stuff which has nothing to do with why someone would "inplace" in the first place, which is because they are happy with their exploratory edits and now explicitly want to change the df for good.

@ReuvenLerner Жыл бұрын

Sorry it didn't work for you; I hadn't thought about that use case, where people don't use inplace=True to explore, and then do use it as a "final" way of doing things. The core Pandas developers have warned us for several years now that inplace=True is going away, and that we should avoid using it, regardless of our motivations. Method chaining is the preferred way to do things, both when exploring and when we finish.

@lade_edal Жыл бұрын

@@ReuvenLerner Roger that. Thanks for the warning :)