How do I avoid a SettingWithCopyWarning in pandas?

If you've been using pandas for a while, you've likely encountered a SettingWithCopyWarning. The proper response is to modify your code appropriately, not to turn off the warning! In this video, I'll show you two common scenarios in which this warning arises, explain why it's occurring, and then demonstrate how to address it.
SUBSCRIBE to learn data science with Python:
kzbin.info?su...
JOIN the "Data School Insiders" community and receive exclusive rewards:
/ dataschool
== RESOURCES ==
GitHub repository for the series: github.com/justmarkham/pandas...
"copy" documentation: pandas.pydata.org/pandas-docs/...
"loc" documentation: pandas.pydata.org/pandas-docs/...
Longer explanation of SettingWithCopyWarning: tomaugspurger.github.io/modern...
Returning a view versus a copy: pandas.pydata.org/pandas-docs/...
== LET'S CONNECT! ==
Newsletter: www.dataschool.io/subscribe/
Twitter: / justmarkham
Facebook: / datascienceschool
LinkedIn: / justmarkham

[@lalu225]

Super clear, brilliantly explained and articulated. The warning generated as per the second scenario was so mysterious to me and I did not understand the documentation. You video is amazing.

[@dataschool]

Thanks very much for your kind words! I appreciate it!

[@jinwoopark151]

omg. i've googled several times and most suggestions didn't work for me. its really crystal clear and helpful. Thanks a lot!!!

[@dataschool]

That's awesome to hear! :)

[@freenrg888]

7 years later, this helped me. Thank you.

[@dataschool]

You're welcome!

[@joaoostrowski]

I'm watching this series every day when I get some free time between work and university. I've been working on a code and receiving the SettingWithCopyWarning without understanding why even after reading all caveats available in pandas API. Fortunately, you just gave me the trick! Adding the .copy() I solved my warning problems and my Market Basket Analysis is running all smooth! Keep it up Data School, an outstanding job being done here and I'm onboard sharing and spreading the word! Cheers from Hungary!

[@dataschool]

That's great to hear! And, thank you for spreading the word! :)

[@jlpicard7]

I spent hours trying to understand this warning after getting tripped up over it on numerous occasions. Nothing I've found so far has come close to clearly explaining what is going with this message as this video. I'd often find a hack that would get me by, but not clearly understanding the cause guaranteed that I'd run into again (and again, and again...). Thanks so much putting these out. I work mostly in R for my day job, but really enjoy working in Python and am becoming a big fan of Data School. Keep up the great work!

[@dataschool]

Awesome! Thanks so much for your kind words! Nice username by the way :)

[@kzarina29]

Ugh wish I found you sooner! Took me forever to look for solutions! Yours was crystal clear. Thanks xx

[@dataschool]

You're welcome!

[@SupreetBose]

Awesome video especially the last part were the .copy() is explained. I was struggling with warning for past 5 days . Awesome job !!

[@dataschool]

Thanks! Glad the video was helpful to you!

[@haneulkim4902]

I've trying to get hold of this topic and haven't fully understood it until now. Thanks!!

[@dataschool]

Great to hear!

[@shree4641]

This is the only video i found that taught me how to fill/update particular cell in csv file (i.e in second example) Thanks a lot!!!!!....You saved my time & you made my doubts super clear. Also i learned how to update/fill a particular cell in a csv file.

[@dataschool]

Awesome! Glad I could be of help :)

[@nussgackl3149]

I kept running into that warning until I saw your video. Thanks for explaining the reasons in such a clear and concise way!

[@dataschool]

You’re very welcome!

[@kevinsteptoe]

This is the most clear and helpful explanation of this topic I have found. The second example I had been struggling with for a long time. Thank you

[@dataschool]

Glad it was helpful!

[@AlyMakhlouf]

last minutes in the video just saved my day. I've been searching for a solution at google for hours and your video was last resort before changing the away from pandas totally. thank you.

[@dataschool]

Glad it helped!

It was a great, useful, excellent lesson. It shown us very deep concepts related to values with square brackets and direct reference with loc. Thanks you, so much.

[@dataschool]

You're very welcome! Glad it was helpful to you!

[@sandaimehokage89]

Dude! I'm just starting to learn Pandas I was getting this warning with the very first lines I wrote. Thank you so much for the clarification.

[@dataschool]

You're very welcome!

[@tommonks2490]

Excellent explanation. This really helped me wrap my head around the SettingWithCopyWarning. Thank you so much!

[@dataschool]

You're very welcome!

[@joey5069]

your channel is the best! Thanks a lot, you made it so clear in such a short video!

[@dataschool]

Thank you!

[@Leonardo-jv1ls]

i haven't started the video yet. But i already loved so much. Thank you once again.

[@dataschool]

You are so kind! 👍

[@Chiara-et1kf]

Thank you very much for your clear and brilliant explanation! It helped me to modify my code and fix the warning.

[@dataschool]

You're very welcome!

[@polakpuckstopper]

Fantastic explanation. Keep up the great work!

[@dataschool]

Thank you!

[@sub.quran.recitations]

Clear and concise explanation! Loved it ❤

[@dataschool]

Glad you liked it!

[@robertue1]

Thank you Kevin, great and super clear video!

[@dataschool]

Glad it was helpful!

[@mohakmehta6929]

Thank you, very much for this video, I was stuck on assignment and this helped me to solve the problem. Brilliantly explained.

[@dataschool]

Great to hear!

[@Edwin-cc8fy]

Great examples! Much better than reading the docs.

[@AustrianAnalyst]

Thank you for this. Great content as always.

[@dataschool]

Much appreciated!

[@ehsanmo1367]

Thanks for the nice explanations! keep up the great work.

[@dataschool]

You're very welcome, and thanks for watching!

[@yashpratapwar6490]

Truly crystal clear! Thanks for the awesome explanation!!

[@dataschool]

You're very welcome! 🙏

[@RajaseelanGaneswaran]

Brilliant explanation. Subscribed!

[@dataschool]

Thanks!

[@janoxakes]

Thanks, this is a very clear explanation. I have been trying to remove the false warnings for some time no knowing what the issue was. I'll test right away!

[@dataschool]

Great to hear! This is one of the most confusing areas of pandas, so it took me a while to truly understand it as well.

[@laxmivanam8630]

Thank you! great video. I am still not clear on one part. Can you please tell me the use of pd.options.mode.chained_assignment = None?? Does it avoid the change??

[@xml6576]

Thanks a lot! This thing just drove me crazy!

[@dataschool]

Glad I could help!

[@pranavdhuri2710]

Finally I got the solution to the problem. Thanks for the clear and concise explaination

[@dataschool]

You're welcome!

[@gunnarsamuelson9042]

Very succinct, the explanation is spot on!

[@dataschool]

Thank you!

[@sleepdeep5230]

Good tutorials never gets old thanks a lot

[@dataschool]

Thanks!

[@sergeiklimkov4136]

very useful stuff, please keep it coming

[@dataschool]

Thanks, will do!

[@rusiano-kq9ws]

Thank you a lot! Super clear as always.

[@dataschool]

You're welcome!

[@emmanouil2586]

It's not too often what the quality of a vid makes me pause 3 minutes into it to like it

[@dataschool]

Thank you! 🙏

[@russellwelsh9663]

Brilliant explanation ... concise and clear.

[@dataschool]

Thanks!

[@vipul5340]

I have been reading documentations for 2 hours and didn't completely get it. Should have known to directly come here. Thank you.

[@silverstreet7984]

This was a life saver. Thank you very much

[@dataschool]

You're very welcome!

[@Tasjemlies_]

thank you so much for the clear explanation! much appreciated !

[@user-sd7zj7zu2n]

Thank you so much! it has a lot of benefits for me!

[@dataschool]

You're welcome!

[@luizpaulosousa7352]

Thank you so much! The second scenario is really a lot confusing.

[@omarboutari8864]

Very helpful and informative.

[@dataschool]

Thank you!

[@jandersonaraujo7796]

Amazing content! It's much clearer to me. Everyone needs to watch this video! You deserve 100.000 likes Conteúdo incrível!

[@dataschool]

Thanks very much for your kind words!

[@Niconico2]

I hate this warning haha, thanks for your help, i was trying to understand why I was receiving that warning and i couldn't understand until watch your video. thx soo much!

[@dataschool]

You're very welcome!

[@cyberdudecyber]

Great explanation. Thank you!

[@dataschool]

You're welcome!

[@amityadav-tu4hc]

super clear explanation.

[@dataschool]

Thanks!

[@Dexter01]

Thank you, it was awesome as always!

[@dataschool]

Thank you!

[@KatHanner]

Hi. Thanks for the video. It clarifies the pandas issue well. However, just a note, a “Not Rated” film is a valuable rating classification as it means that the film was not submitted for a rating by the Motion Picture Association. Personally, when I see “Not Rated” on a film, it conveys that there was a choice made to not have it rated and it’s probably to keep the creative integrity. So, dropping these values may not always be the best approach when you’re building insights from the data. (note: you did say, “for the purpose of this video”) Again though, I really appreciate the example and help deciphering the warning message.

[@dataschool]

Thank you so much for sharing, I did not know that!

[@akshaywayse5274]

Very Very Thank you so much ...this warning was just mind blowing shitt...Once again Thank you 😍

[@Vdherrlichkeit]

excellent ! You Sir, made my day.

[@bosra11]

Excellent video 👌

[@dataschool]

Thank you!

[@Prinard3]

Solved my issue, thank you!

[@dataschool]

Great to hear!

[@omarabouhassan6673]

You're the best! Thank you :)

[@dataschool]

You're welcome!

[@jacobhicks9629]

Watching this in 2022... still helping people :)

[@dataschool]

Great to hear!

[@ZifanYu]

Thank you for the video, now I understand that why I use loc but still warns me.

[@dataschool]

You're welcome!

[@ottobis08]

Lifesaver. Thanks!

[@dataschool]

You're welcome!

[@jasonplewinski2443]

Amazing - thank you!

[@dataschool]

You're very welcome!

[@asadrauf7111]

Wow, you are the best Kevin. Any plans of making a magical video series on how you compete in a kaggle competition?

[@dataschool]

That's under consideration for the future! :)

[@asadrauf7111]

@@dataschool Perfect! Thanks :)

[@johnmartin4105]

Great tutorial, so many code samples for beginners don't cover this and when you're trying to do seemingly simple stuff, these warnings can be off-putting to a new programmer!

[@dataschool]

Glad it was helpful!

[@yuqianxia1140]

Thanks a lot!!!

[@gravitycuda]

Helped me solve my problem :)

[@apostolosmavropoulos177]

best explanation!

[@dataschool]

Thanks!

[@wilmarperez4962]

Outstanding explanation. Thanks a lot! For the first example, I guess this is another reason to prefer using '.loc[]' instead of 'df[]' for selecting/filtering... Am I right?

[@peatahandy3152]

Thanks a lot

[@viktorkutsyi8054]

Thank You!

[@dataschool]

You're welcome!

[@nileshsuryavanshi8792]

Thanks a million, dude.👏👏

[@dataschool]

You're welcome!

[@ssymmetri]

Thanks a lot! I get it now =]

[@dataschool]

Great to hear!

[@AManKumar-pz6et]

for second problem we can use like below form for faster and without warning. top_movies.set_value(0, 'duration', 150) Thanks Data School.

[@dataschool]

I had never heard of set_value! Looks like it has been deprecated, starting with version 0.21.0. So, I would recommend you don't use it any more.

[@FabioRBelotto]

I have been watching you videos on your preset order but I didn't remember to see discussions about copy or views of dataframes

[@vishavgupta3717]

Thank you

[@dataschool]

You're welcome!

[@mojganhtn6360]

thanks man you save my life :)

[@dataschool]

Thank you!

[@ganeshbala6129]

Beautiful!

[@dataschool]

Thank you! Cheers!

[@ramleo1461]

Hi, your videos series on pandas is excellent but please make videos on numpy as well.. Also are your video series on scikit learn enough? Or can u suggest good video series on scikit learn

[@dataschool]

Thanks for your NumPy suggestion! As for whether my scikit-learn videos are "enough", I recommend watching all 7 hours and then deciding 😄 And if you need more, I have a Machine Learning course with 14 more hours: www.dataschool.io/learn/

[@mountainscott5274]

from what I see, another option at 4:30 could be: movies.content_rating.replace("NOT RATED", np.nan, inplace=True) (no SWC warning)

[@dataschool]

Right! Thanks for sharing!

[@krishj8011]

Awesome... thanks

[@dataschool]

You're welcome!

[@cesarshimura8913]

Thank you.

[@dataschool]

You're welcome!

[@PaulSmith-zs5je]

Quality Content...

[@dataschool]

Thanks!

[@haneulkim4902]

Thanks for the video. So movies[movies.content_rating == "NOT RATED"].copy().content_rating = np.nan would work as well since you specifically tell pandas to copy the original df and use that copied df?

[@dataschool]

I wouldn't chain anything after the copy method, or assign anything after the copy method.

[@melvin9993]

Very effective!

[@dataschool]

Thanks!

[@zhonghuahe3556]

i still have a question, in the second example, if we explicitly use .copy() to tell Pandas, top_movies is a copy not a view, the total RAM consumption is higher or not?

[@zhonghuahe3556]

found the answer in this link: realpython.com/pandas-settingwithcopywarning/, .copy() has a parameter that determines if you want a view (deep=False) or copy (deep=True). deep is True by default

[@kptib1988]

wow so clear..

[@dataschool]

Glad you liked it!

[@vishavgupta3717]

You are awesome

[@dataschool]

Thank you!

[@raniphore]

In cases where i don't have the memory to create a copy of the dataframe, how to tackle this error ?

[@johnzhang5114]

very good explanation.!!! but how come pandas doesn't know if it is a view or a copy?

[@ninjaturtle205]

what happens when you want to force return a view? For example: I want to filter some data out of movies, like you do, then i want to apply another filter, and then makes changes on that, , but also have the change be reflected in the original dataframe?

[@dataschool]

If you want to change the original DataFrame, you would use a single loc command with multiple filter criteria. This video might be helpful to you: kzbin.info/www/bejne/j4GspZmHbZykoK8

[@janardhananar7908]

I am getting this error while trying to add a new column to a dataframe within a columntransformer. X.loc[:,AMT_DIFF'] = X.loc[self.dif_col[0]] - X.loc[self.dif_col[1]]. How do I avoid this ?

[@abenazus943]

Hi! very useful ! thanks. I have a question for you, if you can answer. I have 2 dataframe and I would like to copy only one line of their 2 dataframes into a third one. If I used ike you the .copy(), I only have into the third dataframe the latest line and not 2 lines. How can I could proceed? Thanks for your video :)

[@dataschool]

I'm not sure I fully understand, sorry! However, you might want to ask it on Stack Overflow. Here's a guide I wrote for asking good questions: www.dataschool.io/how-to-ask-for-coding-help-online/ Good luck!

[@Ubadidoo]

How to get understanding when the view or copy come out. what are those cases? what is the logic of having one vs another?

[@dataschool]

You can read more about it here: pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

[@atiaspire]

i am still getting error with the below code; please suggest col = ['text'] for col in cols: text_data = df[col].copy() text_data = text_data.apply(remove_punctuations) df.loc[:,col] = text_data print(col)

[@dataschool]

Sorry, I won't be able to help you troubleshoot your code... good luck!

[@costastasoulas8058]

How do I use the .loc solution for the warning when the df[:,ColumnName] is a value from a map?

[@dataschool]

Sorry, I don't quite understand, could you clarify? Thanks!

[@takocchii]

great!

[@dataschool]

Thanks!

[@tahiliani22]

Could someone please shed some light on this ?? In summary, to avoid the first warning we need for it to be a view and not a copy but for the second example, we need for the new_df to be a copy?

[@MK-xm2nh]

Just watching, and I think in the explanation the terms were reversed. A copy is a new dataframe, a view is just getting info from the dataframe - so in the first warning, we also needed it to be a copy, not a view.

[@kishanlal676]

Is it ok to use ' movies.content_rating = movies.content_rating.replace ('NOT RATED',np.nan) ' rather than using ' movies.loc[movies.content_rating=='NOT RATED' , 'content_rating' ] = np.nan ' for replacing 'NOT RATED' with NaN values??? Thanks a lot!!

[@dataschool]

I think it should be fine, but off-hand I'm not positive.

[@mariofishman]

Hi. Great videos! I use copy() if I do not want to change the original dataframe. How do I change the original DataFrame withouth getting the warning? If I don't use copy I get the warning, if I use copy I only modify the copy but not the original. Thanks for your help!

[@dataschool]

Glad you like the videos! Regarding your question, I can't answer it without knowing what code you are writing that is resulting in the warning.

[@1234zztechman]

Great. Could you also please clarify on how to avoid SettingWithCopyWarning on downcast=downcast ,**kwargs

[@dataschool]

Sorry, I don't quite understand your question. Could you clarify?

[@shashisharma3768]

Sir, When u r calling head of df how it comes with border. Pls explain

[@dataschool]

The formatting varies depending on your IDE and IDE version. I'm using Jupyter notebook in the video. Hope that helps!

[@meghasyam427]

Ran into a problem and noticed that .copy and .copy() are two different things and not sure what .copy is doing. when i try to assign top_movies with .copy suffix then top_movies is created as a function. Couldn't find anything related to it on the internet. Can you help me kevin

[@dataschool]

To use the copy method, you have to include parentheses. Hope that helps!

[@akhil4441]

Hi bro... I came across a different warning and I am not sure about it. "This is separate from the ipykernel package so we can avoid doing imports until" can you please suggest..

[@dataschool]

Sorry, I'm not sure what the problem is. Good luck!

[@arunavadatta7734]

I have some Setting With Copy Warning in one of my program that I am not able to solve. Program is running good and giving expected result. But lots of same warnings. I have tried ".loc" but getting wrong answer. Can you please help me???

[@jiayangcheng]

820：1 like/dislike ratio. What a GOAT. Thank you!

[@dataschool]

You're welcome!