How do I avoid a SettingWithCopyWarning in pandas?

[@lalu225]

Super clear, brilliantly explained and articulated. The warning generated as per the second scenario was so mysterious to me and I did not understand the documentation. You video is amazing.

[@dataschool]

Thanks very much for your kind words! I appreciate it!

[@jinwoopark151]

omg. i've googled several times and most suggestions didn't work for me. its really crystal clear and helpful. Thanks a lot!!!

[@dataschool]

That's awesome to hear! :)

[@jlpicard7]

I spent hours trying to understand this warning after getting tripped up over it on numerous occasions. Nothing I've found so far has come close to clearly explaining what is going with this message as this video. I'd often find a hack that would get me by, but not clearly understanding the cause guaranteed that I'd run into again (and again, and again...). Thanks so much putting these out. I work mostly in R for my day job, but really enjoy working in Python and am becoming a big fan of Data School. Keep up the great work!

[@dataschool]

Awesome! Thanks so much for your kind words! Nice username by the way :)

[@cosmopolitanPJ]

I'm watching this series every day when I get some free time between work and university. I've been working on a code and receiving the SettingWithCopyWarning without understanding why even after reading all caveats available in pandas API. Fortunately, you just gave me the trick! Adding the .copy() I solved my warning problems and my Market Basket Analysis is running all smooth! Keep it up Data School, an outstanding job being done here and I'm onboard sharing and spreading the word! Cheers from Hungary!

[@dataschool]

That's great to hear! And, thank you for spreading the word! :)

[@AlyMakhlouf]

last minutes in the video just saved my day. I've been searching for a solution at google for hours and your video was last resort before changing the away from pandas totally. thank you.

[@dataschool]

Glad it helped!

[@haneulkim4902]

I've trying to get hold of this topic and haven't fully understood it until now. Thanks!!

[@dataschool]

Great to hear!

[@nussgackl3149]

I kept running into that warning until I saw your video. Thanks for explaining the reasons in such a clear and concise way!

[@dataschool]

You’re very welcome!

[@SupreetBose]

Awesome video especially the last part were the .copy() is explained. I was struggling with warning for past 5 days . Awesome job !!

[@dataschool]

Thanks! Glad the video was helpful to you!

[@freenrg888]

7 years later, this helped me. Thank you.

[@dataschool]

You're welcome!

[@shree4641]

This is the only video i found that taught me how to fill/update particular cell in csv file (i.e in second example) Thanks a lot!!!!!....You saved my time & you made my doubts super clear. Also i learned how to update/fill a particular cell in a csv file.

[@dataschool]

Awesome! Glad I could be of help :)

[@kzarina29]

Ugh wish I found you sooner! Took me forever to look for solutions! Yours was crystal clear. Thanks xx

[@dataschool]

You're welcome!

[@kevinsteptoe]

This is the most clear and helpful explanation of this topic I have found. The second example I had been struggling with for a long time. Thank you

[@dataschool]

Glad it was helpful!

[@KatHanner]

Hi. Thanks for the video. It clarifies the pandas issue well. However, just a note, a “Not Rated” film is a valuable rating classification as it means that the film was not submitted for a rating by the Motion Picture Association. Personally, when I see “Not Rated” on a film, it conveys that there was a choice made to not have it rated and it’s probably to keep the creative integrity. So, dropping these values may not always be the best approach when you’re building insights from the data. (note: you did say, “for the purpose of this video”) Again though, I really appreciate the example and help deciphering the warning message.

[@dataschool]

Thank you so much for sharing, I did not know that!

[@sandaimehokage89]

Dude! I'm just starting to learn Pandas I was getting this warning with the very first lines I wrote. Thank you so much for the clarification.

[@dataschool]

You're very welcome!

[@theraizadatalks14]

Crystal and clear explanations, thanks for creating this series. This Pandas series helped in my daily work life. Much love ❤️

[@dataschool]

Great to hear!

[@vipul5340]

I have been reading documentations for 2 hours and didn't completely get it. Should have known to directly come here. Thank you.

It was a great, useful, excellent lesson. It shown us very deep concepts related to values with square brackets and direct reference with loc. Thanks you, so much.

[@dataschool]

You're very welcome! Glad it was helpful to you!

[@Leonardo-jv1ls]

i haven't started the video yet. But i already loved so much. Thank you once again.

[@dataschool]

You are so kind! 👍

[@mountainscott5274]

from what I see, another option at 4:30 could be: movies.content_rating.replace("NOT RATED", np.nan, inplace=True) (no SWC warning)

[@dataschool]

Right! Thanks for sharing!

[@sub.quran.recitations]

Clear and concise explanation! Loved it ❤

[@dataschool]

Glad you liked it!

[@polakpuckstopper]

Fantastic explanation. Keep up the great work!

[@dataschool]

Thank you!

[@lixiang7349]

we've written not wrote 12:09

[@janardhananar7908]

I am getting this error while trying to add a new column to a dataframe within a columntransformer. X.loc[:,AMT_DIFF'] = X.loc[self.dif_col[0]] - X.loc[self.dif_col[1]]. How do I avoid this ?

[@pranavdhuri2710]

Finally I got the solution to the problem. Thanks for the clear and concise explaination

[@dataschool]

You're welcome!

[@zXarianCzar]

Truly crystal clear! Thanks for the awesome explanation!!

[@dataschool]

You're very welcome! 🙏

[@tommonks2490]

Excellent explanation. This really helped me wrap my head around the SettingWithCopyWarning. Thank you so much!

[@dataschool]

You're very welcome!

[@mohakmehta6929]

Thank you, very much for this video, I was stuck on assignment and this helped me to solve the problem. Brilliantly explained.

[@dataschool]

Great to hear!

[@sleepdeep5230]

Good tutorials never gets old thanks a lot

[@dataschool]

Thanks!

[@joey5069]

your channel is the best! Thanks a lot, you made it so clear in such a short video!

[@dataschool]

Thank you!

[@AustrianAnalyst]

Thank you for this. Great content as always.

[@dataschool]

Much appreciated!

[@jandersonaraujo7796]

Amazing content! It's much clearer to me. Everyone needs to watch this video! You deserve 100.000 likes Conteúdo incrível!

[@dataschool]

Thanks very much for your kind words!

[@janoxakes]

Thanks, this is a very clear explanation. I have been trying to remove the false warnings for some time no knowing what the issue was. I'll test right away!

[@dataschool]

Great to hear! This is one of the most confusing areas of pandas, so it took me a while to truly understand it as well.

[@laxmivanam8630]

Thank you! great video. I am still not clear on one part. Can you please tell me the use of pd.options.mode.chained_assignment = None?? Does it avoid the change??

[@Chiara-et1kf]

Thank you very much for your clear and brilliant explanation! It helped me to modify my code and fix the warning.

[@dataschool]

You're very welcome!

[@RajaseelanGaneswaran]

Brilliant explanation. Subscribed!

[@dataschool]

Thanks!

[@arunavadatta7734]

I have some Setting With Copy Warning in one of my program that I am not able to solve. Program is running good and giving expected result. But lots of same warnings. I have tried ".loc" but getting wrong answer. Can you please help me???

[@johnmartin4105]

Great tutorial, so many code samples for beginners don't cover this and when you're trying to do seemingly simple stuff, these warnings can be off-putting to a new programmer!

[@dataschool]

Glad it was helpful!

[@FabioRBelotto]

I have been watching you videos on your preset order but I didn't remember to see discussions about copy or views of dataframes

[@atiaspire]

i am still getting error with the below code; please suggest col = ['text'] for col in cols: text_data = df[col].copy() text_data = text_data.apply(remove_punctuations) df.loc[:,col] = text_data print(col)

[@dataschool]

Sorry, I won't be able to help you troubleshoot your code... good luck!

[@emmanouil2586]

It's not too often what the quality of a vid makes me pause 3 minutes into it to like it

[@dataschool]

Thank you! 🙏

[@robertue1]

Thank you Kevin, great and super clear video!

[@dataschool]

Glad it was helpful!

[@raniphore]

In cases where i don't have the memory to create a copy of the dataframe, how to tackle this error ?

[@meghasyam427]

Ran into a problem and noticed that .copy and .copy() are two different things and not sure what .copy is doing. when i try to assign top_movies with .copy suffix then top_movies is created as a function. Couldn't find anything related to it on the internet. Can you help me kevin

[@dataschool]

To use the copy method, you have to include parentheses. Hope that helps!

[@Edwin-cc8fy]

Great examples! Much better than reading the docs.

[@zhonghuahe3556]

i still have a question, in the second example, if we explicitly use .copy() to tell Pandas, top_movies is a copy not a view, the total RAM consumption is higher or not?

[@zhonghuahe3556]

found the answer in this link: realpython.com/pandas-settingwithcopywarning/, .copy() has a parameter that determines if you want a view (deep=False) or copy (deep=True). deep is True by default

[@Niconico2]

I hate this warning haha, thanks for your help, i was trying to understand why I was receiving that warning and i couldn't understand until watch your video. thx soo much!

[@dataschool]

You're very welcome!

[@ehsanmo1367]

Thanks for the nice explanations! keep up the great work.

[@dataschool]

You're very welcome, and thanks for watching!

[@haneulkim4902]

Thanks for the video. So movies[movies.content_rating == "NOT RATED"].copy().content_rating = np.nan would work as well since you specifically tell pandas to copy the original df and use that copied df?

[@dataschool]

I wouldn't chain anything after the copy method, or assign anything after the copy method.

[@shashisharma3768]

Sir, When u r calling head of df how it comes with border. Pls explain

[@dataschool]

The formatting varies depending on your IDE and IDE version. I'm using Jupyter notebook in the video. Hope that helps!

[@kishanlal676]

Is it ok to use ' movies.content_rating = movies.content_rating.replace ('NOT RATED',np.nan) ' rather than using ' movies.loc[movies.content_rating=='NOT RATED' , 'content_rating' ] = np.nan ' for replacing 'NOT RATED' with NaN values??? Thanks a lot!!

[@dataschool]

I think it should be fine, but off-hand I'm not positive.

[@abenazus943]

Hi! very useful ! thanks. I have a question for you, if you can answer. I have 2 dataframe and I would like to copy only one line of their 2 dataframes into a third one. If I used ike you the .copy(), I only have into the third dataframe the latest line and not 2 lines. How can I could proceed? Thanks for your video :)

[@dataschool]

I'm not sure I fully understand, sorry! However, you might want to ask it on Stack Overflow. Here's a guide I wrote for asking good questions: www.dataschool.io/how-to-ask-for-coding-help-online/ Good luck!

[@russellwelsh9663]

Brilliant explanation ... concise and clear.

[@dataschool]

Thanks!

[@luizpaulosousa7352]

Thank you so much! The second scenario is really a lot confusing.

[@jacobhicks9629]

Watching this in 2022... still helping people :)

[@dataschool]

Great to hear!

[@johnzhang5114]

very good explanation.!!! but how come pandas doesn't know if it is a view or a copy?

[@costastasoulas8058]

How do I use the .loc solution for the warning when the df[:,ColumnName] is a value from a map?

[@dataschool]

Sorry, I don't quite understand, could you clarify? Thanks!

[@rusiano-kq9ws]

Thank you a lot! Super clear as always.

[@dataschool]

You're welcome!

[@akhil4441]

Hi bro... I came across a different warning and I am not sure about it. "This is separate from the ipykernel package so we can avoid doing imports until" can you please suggest..

[@dataschool]

Sorry, I'm not sure what the problem is. Good luck!

[@wilmarperez4962]

Outstanding explanation. Thanks a lot! For the first example, I guess this is another reason to prefer using '.loc[]' instead of 'df[]' for selecting/filtering... Am I right?

[@ninjaturtle205]

what happens when you want to force return a view? For example: I want to filter some data out of movies, like you do, then i want to apply another filter, and then makes changes on that, , but also have the change be reflected in the original dataframe?

[@dataschool]

If you want to change the original DataFrame, you would use a single loc command with multiple filter criteria. This video might be helpful to you: kzbin.info/www/bejne/j4GspZmHbZykoK8

[@Ubadidoo]

How to get understanding when the view or copy come out. what are those cases? what is the logic of having one vs another?

[@dataschool]

You can read more about it here: pandas.pydata.org/pandas-docs/stable/indexing.html#indexing-view-versus-copy

[@AManKumar-pz6et]

for second problem we can use like below form for faster and without warning. top_movies.set_value(0, 'duration', 150) Thanks Data School.

[@dataschool]

I had never heard of set_value! Looks like it has been deprecated, starting with version 0.21.0. So, I would recommend you don't use it any more.

[@terryliu3635]

Hello, Thanks again for the great videos. I have a question on applying conditional "substring" to a column of the DataFrame. My example DataFrame is very simple, 2 columns: translate and lenofdate. 2018-1-31 XXXXXXXX, 18 2018-10-31 XXXXXXXX, 19 2018-11-1 XXXXXXXX, 18 2018-11-20 XXXXXXXX, 19 I would like to use the 2nd column to "conditionally" take "lenofdate - 9" number of chars from the 1st column with the following code: transdf['transdate'].str[0:(transdf['lenofdate'] - 9)] However, this does not work. I'm wondering if you could give me some helps. Thanks in advance! Terry

[@dataschool]

Hi Terry - I'm sorry, this is a more detailed question than I have time to review. You're welcome to join my membership program, and I'd be happy to take a second look - members are the only people I'm able to invest more time into: www.patreon.com/dataschool Thanks for understanding!

[@roberthromada4029]

Hi Kevin. I have question regarding your video. I've just started to work with pandas. Can you help me how to solve some warning as you presented here for these two lines of code df_DBver3['release_date'] = df_DBver3['release_date'].apply(lambda x: x.split("/")[0]) df_DBver3.rename(columns={'release_date': 'release_month'}, inplace=True) Thx.

[@dataschool]

I won't be able to advise, sorry!

[@ramleo1461]

Hi, your videos series on pandas is excellent but please make videos on numpy as well.. Also are your video series on scikit learn enough? Or can u suggest good video series on scikit learn

[@dataschool]

Thanks for your NumPy suggestion! As for whether my scikit-learn videos are "enough", I recommend watching all 7 hours and then deciding 😄 And if you need more, I have a Machine Learning course with 14 more hours: www.dataschool.io/learn/

[@bosra11]

Excellent video 👌

[@dataschool]

Thank you!

[@cyberdudecyber]

Great explanation. Thank you!

[@dataschool]

You're welcome!

[@gunnarsamuelson9042]

Very succinct, the explanation is spot on!

[@dataschool]

Thank you!

[@raffdoc]

HI, how can one do multiple imputation using pandas library? Thanks beforehand

[@dataschool]

I think you are looking for 'fillna', which I cover in this video: kzbin.info/www/bejne/nHSwo4KVi9-Ygpo Let me know if that doesn't answer your question!

[@raffdoc]

I have already seen all videos from these series. What I really mean is how to input na values using different statistical/probabilistic algorithms (ex. Gibbs sampling eat). Thanks for rapid response.

[@dataschool]

Got it! Sorry, I don't know how to do that... let me know if you figure out!

[@sergeiklimkov4136]

very useful stuff, please keep it coming

[@dataschool]

Thanks, will do!

[@Dexter01]

Thank you, it was awesome as always!

[@dataschool]

Thank you!

[@雍壹钧]

Thank you so much! it has a lot of benefits for me!

[@dataschool]

You're welcome!

[@tahiliani22]

Could someone please shed some light on this ?? In summary, to avoid the first warning we need for it to be a view and not a copy but for the second example, we need for the new_df to be a copy?

[@MK-xm2nh]

Just watching, and I think in the explanation the terms were reversed. A copy is a new dataframe, a view is just getting info from the dataframe - so in the first warning, we also needed it to be a copy, not a view.

[@1234zztechman]

Great. Could you also please clarify on how to avoid SettingWithCopyWarning on downcast=downcast ,**kwargs

[@dataschool]

Sorry, I don't quite understand your question. Could you clarify?

[@xml6576]

Thanks a lot! This thing just drove me crazy!

[@dataschool]

Glad I could help!

[@mariofishman]

Hi. Great videos! I use copy() if I do not want to change the original dataframe. How do I change the original DataFrame withouth getting the warning? If I don't use copy I get the warning, if I use copy I only modify the copy but not the original. Thanks for your help!

[@dataschool]

Glad you like the videos! Regarding your question, I can't answer it without knowing what code you are writing that is resulting in the warning.

[@suhayryz]

Hi Kevin! How can I use pandas dataframe to perform operation of NLTK (such as spellchecking) or textblob (such as translating) to my dataset? Final result will return back to dataframe and save, maybe new name. Any idea to do this? Anyone? Thanks in advance.

[@dataschool]

In general, you can use a Series "apply" method to apply any arbitrary function (such as translation) to a pandas Series, and then save the result as a new Series in the DataFrame. If you have an interest in Natural Language Processing using Python, I have an entire course about it: www.dataschool.io/learn/ I also have a free tutorial video covering machine learning with text: kzbin.info/www/bejne/kJqufnyrjs6Xj5I Hope that helps!

[@suhayryz]

Yes, I'm doing NLP for Sentiment Analysis in foreign language. Okay. I'll look into pandas series. Thank you much for your help.

[@dataschool]

You're welcome!

[@akshaywayse5274]

Very Very Thank you so much ...this warning was just mind blowing shitt...Once again Thank you 😍

[@omarboutari8864]

Very helpful and informative.

[@dataschool]

Thank you!

[@Tasjemlies_]

thank you so much for the clear explanation! much appreciated !

[@silverstreet7984]

This was a life saver. Thank you very much

[@dataschool]

You're very welcome!

[@asadrauf7111]

Wow, you are the best Kevin. Any plans of making a magical video series on how you compete in a kaggle competition?

[@dataschool]

That's under consideration for the future! :)

[@asadrauf7111]

@@dataschool Perfect! Thanks :)

[@amityadav-tu4hc]

super clear explanation.

[@dataschool]

Thanks!

[@alchemication]

Thanks for another great video Kevin! Do you know if you can do the inverse to what you did at 12:28 (i.e. tell Pandas - this is a reference and I want to update original data frame as well? ;) I would strongly advice to anyone to never leave warnings inside your code. If code with warnings is called from another programming language (read semi-automating data science) - Python might return an error code back and your scripts might fail. Just a heads up as it happened to me and caused headaches ;)

[@dataschool]

That's a great question, and I think the answer is "no": You can't explicitly instruct pandas to create a reference instead of a copy, such that a warning won't be generated.

[@nileshsuryavanshi8792]

Thanks a million, dude.👏👏

[@dataschool]

You're welcome!

[@ZifanYu]

Thank you for the video, now I understand that why I use loc but still warns me.

[@dataschool]

You're welcome!

[@Prinard3]

Solved my issue, thank you!

[@dataschool]

Great to hear!

[@Vdherrlichkeit]

excellent ! You Sir, made my day.

[@omarabouhassan6673]

You're the best! Thank you :)

[@dataschool]

You're welcome!

[@ottobis08]

Lifesaver. Thanks!

[@dataschool]

You're welcome!

[@binwang3241]

does it mean as long as we don't get the warning, all our coding will be fine?

[@dataschool]

Python can't tell you whether the code you are writing does what you intended it to do - it can only tell you whether you are breaking the rules! :)

[@apostolosmavropoulos177]

best explanation!

[@dataschool]

Thanks!

[@mojganhtn6360]

thanks man you save my life :)

[@dataschool]

Thank you!

[@artyb3115]

Thanks a lot, but wtf no bonus today? 😀

[@dataschool]

🤷‍♂️

[@jasonplewinski2443]

Amazing - thank you!

[@dataschool]

You're very welcome!

[@ssymmetri]

Thanks a lot! I get it now =]

[@dataschool]

Great to hear!

[@yuqianxia1140]

Thanks a lot!!!

[@gravitycuda]

Helped me solve my problem :)

[@PaulSmith-zs5je]

Quality Content...

[@dataschool]

Thanks!

[@viktorkutsyi8054]

Thank You!

[@dataschool]

You're welcome!

[@ganeshbala6129]

Beautiful!

[@dataschool]

Thank you! Cheers!