How do I avoid a SettingWithCopyWarning in pandas?

[@lalu225]

Super clear, brilliantly explained and articulated. The warning generated as per the second scenario was so mysterious to me and I did not understand the documentation. You video is amazing.

[@dataschool]

Thanks very much for your kind words! I appreciate it!

[@jinwoopark151]

omg. i've googled several times and most suggestions didn't work for me. its really crystal clear and helpful. Thanks a lot!!!

[@dataschool]

That's awesome to hear! :)

[@AlyMakhlouf]

last minutes in the video just saved my day. I've been searching for a solution at google for hours and your video was last resort before changing the away from pandas totally. thank you.

[@dataschool]

Glad it helped!

[@jlpicard7]

I spent hours trying to understand this warning after getting tripped up over it on numerous occasions. Nothing I've found so far has come close to clearly explaining what is going with this message as this video. I'd often find a hack that would get me by, but not clearly understanding the cause guaranteed that I'd run into again (and again, and again...). Thanks so much putting these out. I work mostly in R for my day job, but really enjoy working in Python and am becoming a big fan of Data School. Keep up the great work!

[@dataschool]

Awesome! Thanks so much for your kind words! Nice username by the way :)

[@cosmopolitanPJ]

I'm watching this series every day when I get some free time between work and university. I've been working on a code and receiving the SettingWithCopyWarning without understanding why even after reading all caveats available in pandas API. Fortunately, you just gave me the trick! Adding the .copy() I solved my warning problems and my Market Basket Analysis is running all smooth! Keep it up Data School, an outstanding job being done here and I'm onboard sharing and spreading the word! Cheers from Hungary!

[@dataschool]

That's great to hear! And, thank you for spreading the word! :)

[@nussgackl3149]

I kept running into that warning until I saw your video. Thanks for explaining the reasons in such a clear and concise way!

[@dataschool]

You’re very welcome!

[@haneulkim4902]

I've trying to get hold of this topic and haven't fully understood it until now. Thanks!!

[@dataschool]

Great to hear!

[@freenrg888]

7 years later, this helped me. Thank you.

[@dataschool]

You're welcome!

[@SupreetBose]

Awesome video especially the last part were the .copy() is explained. I was struggling with warning for past 5 days . Awesome job !!

[@dataschool]

Thanks! Glad the video was helpful to you!

[@shree4641]

This is the only video i found that taught me how to fill/update particular cell in csv file (i.e in second example) Thanks a lot!!!!!....You saved my time & you made my doubts super clear. Also i learned how to update/fill a particular cell in a csv file.

[@dataschool]

Awesome! Glad I could be of help :)

[@kevinsteptoe]

This is the most clear and helpful explanation of this topic I have found. The second example I had been struggling with for a long time. Thank you

[@dataschool]

Glad it was helpful!

[@sandaimehokage89]

Dude! I'm just starting to learn Pandas I was getting this warning with the very first lines I wrote. Thank you so much for the clarification.

[@dataschool]

You're very welcome!

[@kzarina29]

Ugh wish I found you sooner! Took me forever to look for solutions! Yours was crystal clear. Thanks xx

[@dataschool]

You're welcome!

[@theraizadatalks14]

Crystal and clear explanations, thanks for creating this series. This Pandas series helped in my daily work life. Much love ❤️

[@dataschool]

Great to hear!

[@Leonardo-jv1ls]

i haven't started the video yet. But i already loved so much. Thank you once again.

[@dataschool]

You are so kind! 👍

[@vipul5340]

I have been reading documentations for 2 hours and didn't completely get it. Should have known to directly come here. Thank you.

It was a great, useful, excellent lesson. It shown us very deep concepts related to values with square brackets and direct reference with loc. Thanks you, so much.

[@dataschool]

You're very welcome! Glad it was helpful to you!

[@mountainscott5274]

from what I see, another option at 4:30 could be: movies.content_rating.replace("NOT RATED", np.nan, inplace=True) (no SWC warning)

[@dataschool]

Right! Thanks for sharing!

[@KatHanner]

Hi. Thanks for the video. It clarifies the pandas issue well. However, just a note, a “Not Rated” film is a valuable rating classification as it means that the film was not submitted for a rating by the Motion Picture Association. Personally, when I see “Not Rated” on a film, it conveys that there was a choice made to not have it rated and it’s probably to keep the creative integrity. So, dropping these values may not always be the best approach when you’re building insights from the data. (note: you did say, “for the purpose of this video”) Again though, I really appreciate the example and help deciphering the warning message.

[@dataschool]

Thank you so much for sharing, I did not know that!

[@pranavdhuri2710]

Finally I got the solution to the problem. Thanks for the clear and concise explaination

[@dataschool]

You're welcome!

[@mohakmehta6929]

Thank you, very much for this video, I was stuck on assignment and this helped me to solve the problem. Brilliantly explained.

[@dataschool]

Great to hear!

[@sub.quran.recitations]

Clear and concise explanation! Loved it ❤

[@dataschool]

Glad you liked it!

[@tommonks2490]

Excellent explanation. This really helped me wrap my head around the SettingWithCopyWarning. Thank you so much!

[@dataschool]

You're very welcome!

[@janoxakes]

Thanks, this is a very clear explanation. I have been trying to remove the false warnings for some time no knowing what the issue was. I'll test right away!

[@dataschool]

Great to hear! This is one of the most confusing areas of pandas, so it took me a while to truly understand it as well.

[@laxmivanam8630]

Thank you! great video. I am still not clear on one part. Can you please tell me the use of pd.options.mode.chained_assignment = None?? Does it avoid the change??

[@jandersonaraujo7796]

Amazing content! It's much clearer to me. Everyone needs to watch this video! You deserve 100.000 likes Conteúdo incrível!

[@dataschool]

Thanks very much for your kind words!

[@emmanouil2586]

It's not too often what the quality of a vid makes me pause 3 minutes into it to like it

[@dataschool]

Thank you! 🙏

[@zXarianCzar]

Truly crystal clear! Thanks for the awesome explanation!!

[@dataschool]

You're very welcome! 🙏

[@sleepdeep5230]

Good tutorials never gets old thanks a lot

[@dataschool]

Thanks!

[@Chiara-et1kf]

Thank you very much for your clear and brilliant explanation! It helped me to modify my code and fix the warning.

[@dataschool]

You're very welcome!

[@Edwin-cc8fy]

Great examples! Much better than reading the docs.

[@Niconico2]

I hate this warning haha, thanks for your help, i was trying to understand why I was receiving that warning and i couldn't understand until watch your video. thx soo much!

[@dataschool]

You're very welcome!

[@johnmartin4105]

Great tutorial, so many code samples for beginners don't cover this and when you're trying to do seemingly simple stuff, these warnings can be off-putting to a new programmer!

[@dataschool]

Glad it was helpful!

[@polakpuckstopper]

Fantastic explanation. Keep up the great work!

[@dataschool]

Thank you!

[@joey5069]

your channel is the best! Thanks a lot, you made it so clear in such a short video!

[@dataschool]

Thank you!

[@AustrianAnalyst]

Thank you for this. Great content as always.

[@dataschool]

Much appreciated!

[@robertue1]

Thank you Kevin, great and super clear video!

[@dataschool]

Glad it was helpful!

[@RajaseelanGaneswaran]

Brilliant explanation. Subscribed!

[@dataschool]

Thanks!

[@russellwelsh9663]

Brilliant explanation ... concise and clear.

[@dataschool]

Thanks!

[@luizpaulosousa7352]

Thank you so much! The second scenario is really a lot confusing.

[@AManKumar-pz6et]

for second problem we can use like below form for faster and without warning. top_movies.set_value(0, 'duration', 150) Thanks Data School.

[@dataschool]

I had never heard of set_value! Looks like it has been deprecated, starting with version 0.21.0. So, I would recommend you don't use it any more.

[@janardhananar7908]

I am getting this error while trying to add a new column to a dataframe within a columntransformer. X.loc[:,AMT_DIFF'] = X.loc[self.dif_col[0]] - X.loc[self.dif_col[1]]. How do I avoid this ?

[@FabioRBelotto]

I have been watching you videos on your preset order but I didn't remember to see discussions about copy or views of dataframes

[@jacobhicks9629]

Watching this in 2022... still helping people :)

[@dataschool]

Great to hear!

[@wilmarperez4962]

Outstanding explanation. Thanks a lot! For the first example, I guess this is another reason to prefer using '.loc[]' instead of 'df[]' for selecting/filtering... Am I right?

[@arunavadatta7734]

I have some Setting With Copy Warning in one of my program that I am not able to solve. Program is running good and giving expected result. But lots of same warnings. I have tried ".loc" but getting wrong answer. Can you please help me???

[@ehsanmo1367]

Thanks for the nice explanations! keep up the great work.

[@dataschool]

You're very welcome, and thanks for watching!

[@zhonghuahe3556]

i still have a question, in the second example, if we explicitly use .copy() to tell Pandas, top_movies is a copy not a view, the total RAM consumption is higher or not?

[@zhonghuahe3556]

found the answer in this link: realpython.com/pandas-settingwithcopywarning/, .copy() has a parameter that determines if you want a view (deep=False) or copy (deep=True). deep is True by default

[@rusiano-kq9ws]

Thank you a lot! Super clear as always.

[@dataschool]

You're welcome!

[@atiaspire]

i am still getting error with the below code; please suggest col = ['text'] for col in cols: text_data = df[col].copy() text_data = text_data.apply(remove_punctuations) df.loc[:,col] = text_data print(col)

[@dataschool]

Sorry, I won't be able to help you troubleshoot your code... good luck!

[@raniphore]

In cases where i don't have the memory to create a copy of the dataframe, how to tackle this error ?

[@雍壹钧]

Thank you so much! it has a lot of benefits for me!

[@dataschool]

You're welcome!

[@gunnarsamuelson9042]

Very succinct, the explanation is spot on!

[@dataschool]

Thank you!

[@xml6576]

Thanks a lot! This thing just drove me crazy!

[@dataschool]

Glad I could help!

[@meghasyam427]

Ran into a problem and noticed that .copy and .copy() are two different things and not sure what .copy is doing. when i try to assign top_movies with .copy suffix then top_movies is created as a function. Couldn't find anything related to it on the internet. Can you help me kevin

[@dataschool]

To use the copy method, you have to include parentheses. Hope that helps!

[@haneulkim4902]

Thanks for the video. So movies[movies.content_rating == "NOT RATED"].copy().content_rating = np.nan would work as well since you specifically tell pandas to copy the original df and use that copied df?

[@dataschool]

I wouldn't chain anything after the copy method, or assign anything after the copy method.

[@kishanlal676]

Is it ok to use ' movies.content_rating = movies.content_rating.replace ('NOT RATED',np.nan) ' rather than using ' movies.loc[movies.content_rating=='NOT RATED' , 'content_rating' ] = np.nan ' for replacing 'NOT RATED' with NaN values??? Thanks a lot!!

[@dataschool]

I think it should be fine, but off-hand I'm not positive.

[@cyberdudecyber]

Great explanation. Thank you!

[@dataschool]

You're welcome!

[@Dexter01]

Thank you, it was awesome as always!

[@dataschool]

Thank you!

[@sergeiklimkov4136]

very useful stuff, please keep it coming

[@dataschool]

Thanks, will do!

[@shashisharma3768]

Sir, When u r calling head of df how it comes with border. Pls explain

[@dataschool]

The formatting varies depending on your IDE and IDE version. I'm using Jupyter notebook in the video. Hope that helps!

[@tahiliani22]

Could someone please shed some light on this ?? In summary, to avoid the first warning we need for it to be a view and not a copy but for the second example, we need for the new_df to be a copy?

[@MK-xm2nh]

Just watching, and I think in the explanation the terms were reversed. A copy is a new dataframe, a view is just getting info from the dataframe - so in the first warning, we also needed it to be a copy, not a view.

[@ninjaturtle205]

what happens when you want to force return a view? For example: I want to filter some data out of movies, like you do, then i want to apply another filter, and then makes changes on that, , but also have the change be reflected in the original dataframe?

[@dataschool]

If you want to change the original DataFrame, you would use a single loc command with multiple filter criteria. This video might be helpful to you: kzbin.info/www/bejne/j4GspZmHbZykoK8

[@costastasoulas8058]

How do I use the .loc solution for the warning when the df[:,ColumnName] is a value from a map?

[@dataschool]

Sorry, I don't quite understand, could you clarify? Thanks!

[@Ubadidoo]

How to get understanding when the view or copy come out. what are those cases? what is the logic of having one vs another?

[@dataschool]

You can read more about it here: pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

[@abenazus943]

Hi! very useful ! thanks. I have a question for you, if you can answer. I have 2 dataframe and I would like to copy only one line of their 2 dataframes into a third one. If I used ike you the .copy(), I only have into the third dataframe the latest line and not 2 lines. How can I could proceed? Thanks for your video :)

[@dataschool]

I'm not sure I fully understand, sorry! However, you might want to ask it on Stack Overflow. Here's a guide I wrote for asking good questions: www.dataschool.io/how-to-ask-for-coding-help-online/ Good luck!

[@Prinard3]

Solved my issue, thank you!

[@dataschool]

Great to hear!

[@akhil4441]

Hi bro... I came across a different warning and I am not sure about it. "This is separate from the ipykernel package so we can avoid doing imports until" can you please suggest..

[@dataschool]

Sorry, I'm not sure what the problem is. Good luck!

[@silverstreet7984]

This was a life saver. Thank you very much

[@dataschool]

You're very welcome!

[@Tasjemlies_]

thank you so much for the clear explanation! much appreciated !

[@bosra11]

Excellent video 👌

[@dataschool]

Thank you!

[@terryliu3635]

Hello, Thanks again for the great videos. I have a question on applying conditional "substring" to a column of the DataFrame. My example DataFrame is very simple, 2 columns: translate and lenofdate. 2018-1-31 XXXXXXXX, 18 2018-10-31 XXXXXXXX, 19 2018-11-1 XXXXXXXX, 18 2018-11-20 XXXXXXXX, 19 I would like to use the 2nd column to "conditionally" take "lenofdate - 9" number of chars from the 1st column with the following code: transdf['transdate'].str[0:(transdf['lenofdate'] - 9)] However, this does not work. I'm wondering if you could give me some helps. Thanks in advance! Terry

[@dataschool]

Hi Terry - I'm sorry, this is a more detailed question than I have time to review. You're welcome to join my membership program, and I'd be happy to take a second look - members are the only people I'm able to invest more time into: www.patreon.com/dataschool Thanks for understanding!

[@Vdherrlichkeit]

excellent ! You Sir, made my day.

[@1234zztechman]

Great. Could you also please clarify on how to avoid SettingWithCopyWarning on downcast=downcast ,**kwargs

[@dataschool]

Sorry, I don't quite understand your question. Could you clarify?

[@ZifanYu]

Thank you for the video, now I understand that why I use loc but still warns me.

[@dataschool]

You're welcome!

[@amityadav-tu4hc]

super clear explanation.

[@dataschool]

Thanks!

[@roberthromada4029]

Hi Kevin. I have question regarding your video. I've just started to work with pandas. Can you help me how to solve some warning as you presented here for these two lines of code df_DBver3['release_date'] = df_DBver3['release_date'].apply(lambda x: x.split("/")[0]) df_DBver3.rename(columns={'release_date': 'release_month'}, inplace=True) Thx.

[@dataschool]

I won't be able to advise, sorry!

[@nileshsuryavanshi8792]

Thanks a million, dude.👏👏

[@dataschool]

You're welcome!

[@akshaywayse5274]

Very Very Thank you so much ...this warning was just mind blowing shitt...Once again Thank you 😍

[@johnzhang5114]

very good explanation.!!! but how come pandas doesn't know if it is a view or a copy?

[@lixiang7349]

we've written not wrote 12:09

[@ottobis08]

Lifesaver. Thanks!

[@dataschool]

You're welcome!

[@omarboutari8864]

Very helpful and informative.

[@dataschool]

Thank you!

[@raffdoc]

HI, how can one do multiple imputation using pandas library? Thanks beforehand

[@dataschool]

I think you are looking for 'fillna', which I cover in this video: kzbin.info/www/bejne/nHSwo4KVi9-Ygpo Let me know if that doesn't answer your question!

[@raffdoc]

I have already seen all videos from these series. What I really mean is how to input na values using different statistical/probabilistic algorithms (ex. Gibbs sampling eat). Thanks for rapid response.

[@dataschool]

Got it! Sorry, I don't know how to do that... let me know if you figure out!

[@omarabouhassan6673]

You're the best! Thank you :)

[@dataschool]

You're welcome!

[@suhayryz]

Hi Kevin! How can I use pandas dataframe to perform operation of NLTK (such as spellchecking) or textblob (such as translating) to my dataset? Final result will return back to dataframe and save, maybe new name. Any idea to do this? Anyone? Thanks in advance.

[@dataschool]

In general, you can use a Series "apply" method to apply any arbitrary function (such as translation) to a pandas Series, and then save the result as a new Series in the DataFrame. If you have an interest in Natural Language Processing using Python, I have an entire course about it: www.dataschool.io/learn/ I also have a free tutorial video covering machine learning with text: kzbin.info/www/bejne/kJqufnyrjs6Xj5I Hope that helps!

[@suhayryz]

Yes, I'm doing NLP for Sentiment Analysis in foreign language. Okay. I'll look into pandas series. Thank you much for your help.

[@dataschool]

You're welcome!

[@mariofishman]

Hi. Great videos! I use copy() if I do not want to change the original dataframe. How do I change the original DataFrame withouth getting the warning? If I don't use copy I get the warning, if I use copy I only modify the copy but not the original. Thanks for your help!

[@dataschool]

Glad you like the videos! Regarding your question, I can't answer it without knowing what code you are writing that is resulting in the warning.

[@mojganhtn6360]

thanks man you save my life :)

[@dataschool]

Thank you!

[@ssymmetri]

Thanks a lot! I get it now =]

[@dataschool]

Great to hear!

[@apostolosmavropoulos177]

best explanation!

[@dataschool]

Thanks!

[@ramleo1461]

Hi, your videos series on pandas is excellent but please make videos on numpy as well.. Also are your video series on scikit learn enough? Or can u suggest good video series on scikit learn

[@dataschool]

Thanks for your NumPy suggestion! As for whether my scikit-learn videos are "enough", I recommend watching all 7 hours and then deciding 😄 And if you need more, I have a Machine Learning course with 14 more hours: www.dataschool.io/learn/

[@asadrauf7111]

Wow, you are the best Kevin. Any plans of making a magical video series on how you compete in a kaggle competition?

[@dataschool]

That's under consideration for the future! :)

[@asadrauf7111]

@@dataschool Perfect! Thanks :)

[@jasonplewinski2443]

Amazing - thank you!

[@dataschool]

You're very welcome!

[@yuqianxia1140]

Thanks a lot!!!

[@ganeshbala6129]

Beautiful!

[@dataschool]

Thank you! Cheers!

[@viktorkutsyi8054]

Thank You!

[@dataschool]

You're welcome!

[@melvin9993]

Very effective!

[@dataschool]

Thanks!

[@PaulSmith-zs5je]

Quality Content...

[@dataschool]

Thanks!

[@alchemication]

Thanks for another great video Kevin! Do you know if you can do the inverse to what you did at 12:28 (i.e. tell Pandas - this is a reference and I want to update original data frame as well? ;) I would strongly advice to anyone to never leave warnings inside your code. If code with warnings is called from another programming language (read semi-automating data science) - Python might return an error code back and your scripts might fail. Just a heads up as it happened to me and caused headaches ;)

[@dataschool]

That's a great question, and I think the answer is "no": You can't explicitly instruct pandas to create a reference instead of a copy, such that a warning won't be generated.

[@vishavgupta3717]

You are awesome

[@dataschool]

Thank you!

[@gravitycuda]

Helped me solve my problem :)