How do I handle missing values in pandas?

[@dataschool]

In pandas version 0.21 (released October 2017), they added 'isna' and 'notna' as aliases for 'isnull' and 'notnull'. Learn more in my latest video, "5 new changes in pandas you need to know about": kzbin.info/www/bejne/qpaYe6WJeLxggrs

[@bragattemas]

Even in the final of 2019 your material form 2016 still gives incredible help. I have certainty the DataSchool will keep been a success and helping people. Excellent job Kevin Markham. Thanks.

[@Taranggpt6]

Why after replacing na with *various* the count is different . Coubts of various must be equal to na values earlier which was 2644

[@EdgardThreat]

​@@Taranggpt6 hi, that's because there is already a category named "VARIOUS" in the dataset, so the new filled in data gets added up to the existing count of "VARIOUS".

[@vigneshpadmanabhan]

can we get a video on how to handle missing values for data time related datasets. may be sensor values or any sensitive values. multiple varieties of handling missing value would be very useful.

[@nadyamoscow2461]

@@bragattemas I must say even in 2021 it is still completely up to date

[@339059331]

I like his way of teaching, he doesn't assume that the audience knows by default. He breaks down the explanation piece by piece, it is a great learning experience, concise and clear stated lectures as always! Thanks!

[@dataschool]

You're very welcome! Thanks for your kind words!

[@depokboy]

@@dataschool first time watch,,,,those positves comments are true,,,,,thanks a lot

[@codesandroads]

I never leave this place unsatisfied or without answers, total treasure.

[@dataschool]

Thank you so much!

[@BAIBHAVPATHYBEE]

6 years has gone released this video and i m watching it now and it still made me fall in love with the series ... beautifully explained every concept in detail.

[@dataschool]

Thank you so much! 🙏

[@nasserabachi9625]

Now i am in love with Pandas just by seeing a couple of your videos, Shukran Jazeelan !

[@dataschool]

That's awesome! Thanks for sharing!

[@luqikong283]

The most amazing python tutorial I've watched so far. Fell in love with python.

[@rahuldeepdraws8699]

This is actually the most clearly explained video on DataFrames that I have ever come across. Glad I found you. Thank you so much.

[@dataschool]

Glad it was helpful!

[@astroinceptor]

You saved my life twice today, your videos are great and the way you explain is really good. Thank you!

[@dataschool]

Thank you!

[@gabrielreilly7010]

Great videos covering the basics. I enjoy how the additional values within the functions are covered, i.e. axis, etc.

[@dataschool]

Glad it was helpful!

[@kuldipchauhan524]

Awesome- you are gifted.... -- your explanation and content are clean and effective.

[@dataschool]

Thanks very much for your kind words!

[@indreshkumar2002]

u are superb.i took a paid course but they were not able to make me explain these things as u explained me in such a easy way.thnx a lot.

[@dataschool]

You are very welcome! Thanks so much for your kind comment!

[@saachishivhare4836]

I am really loving your videos. Explored your channel just 2 days back!! Earlier I had no idea about pandas but after watching your video, I feel that I will be able to work on my assignment. Great Work! Thank you!

[@stevechops3226]

I cannot tell you how much you have helped me, with all sorts of problems! You have the clearest way of explaining things, thank you so much!

[@dataschool]

You're so very welcome, thanks for your kind words! 🙏

[@kiranachanta9741]

I have been watching Kevin Videos, needless to say he is an Awesome Instructor. His explanation in all of his videos is Conceptual, In-depth and breaking down any complex topic into the easiest way. Thanks Kevin for your great Work!!! It would be great if you could make videos on visualization using Matplotlib & Seaborn.

[@dataschool]

Thanks for your kind words, and for your suggestions! :)

[@sinabaghaei3504]

Your way of teaching makes learning Data Analysis very interesting to me. I really appreciate and wish you success.

[@dataschool]

Thank you!

[@jiaxufan7050]

The best pandas tutorial ever. Hands down.

[@dataschool]

Wow! Thank you so much for your kind comment!

[@grumpyae86]

Exceptional would be a single word to describe your tutorial. Looking forward to binging on your videos lol. Thank you for such clear explanation.

[@dataschool]

Thank you! 🙏

[@TrevorHigbee]

Great videos. I love how all the CSVs are available online.

[@dataschool]

Thanks! 😄

[@FULLCOUNSEL]

You are doing an excellent job. You are called to do this for sure. Cheers

[@dataschool]

Wow, thank you so much for your comment! I really appreciate it.

[@fredcalo]

This was awesome. Clear, concise, incredibly easy to follow. Your explanations (and bonus) were exactly what I was looking for.

[@dataschool]

Excellent! I'm glad the video was helpful to you!

[@firdharamadhani5162]

i rarely leave youtube comment but thank you!! if it werent for your video i wouldn't understand how to do my assignment at all, you did a great job at explaining!

[@TR3NDSETR]

Thanks so much for making this video. You spoke slowly, clearly and very concise. Other videos I have to rewind and watch over, but i dont have to do that here. Looking forward to watching other.

[@dataschool]

That's awesome to hear! Thanks for watching my videos 👍

[@-MinhazulFerdous]

you are a life saver man...... i was fucked up with errors for only 2 missing values in a row of 1000 data

[@dataschool]

Glad to hear I could be of help!

[@mustafabohra2070]

The content you shared is Gold!!

[@dataschool]

Thanks!

[@konradpyrz8559]

This yung gentleman is simply amazing.

[@dataschool]

Thank you! I'm actually 40 years old now 😊

[@rahulbhusari1478]

Really clear and amazing tutorial

[@dataschool]

Glad it was helpful!

[@sammy0722]

Good video. Learnt a lot in short and crisp way

[@dataschool]

Thanks!

[@Person_Not_Known]

Thanks for your videos. most of the python online course i took... i just couldn't get into. Something about your cadence, data sets, and or approach just clicks with me. Thanks for the content.

[@dataschool]

That's awesome! Thanks so much for sharing!

[@msctube45]

Excellent video Data School, very helpful, your explanations are clear and objective. Thank you !

[@vishwanathg8083]

Thank you , You made learning pandas a cake walk.

[@dataschool]

Awesome, that's great to hear!

[@SachinGairola]

great video series, I always fall back here whenever I'm stuck..Thanx for making them so informative...cheers

[@dataschool]

Thanks very much for your kind words!

[@wesleypgurira7142]

and by the way i love the way you teach , its just perfect

[@dataschool]

Thank you!

[@nadyamoscow2461]

Thanks a lot, your course is really helpful and very detailed. You are a great teacher!

[@dataschool]

Thank you!

[@ashishsahu2925]

Really helpful. This means if one needs to figure out number of rows with 1 or more Null values, the code should look like dataframe[dataframe.isnull().sum(axis=1) > 0].

[@eniisy]

Dude it's just an awesome video, forgive me for saying this turning playback speed 1.25 is felts more normal hahah .Love ya, appreciate for your effort about teaching piece by piece !!!!!

[@dataschool]

Thank you!

[@MrsRimouch]

Thank you so much. It is always clearer to listen to you!!

[@dataschool]

You are so welcome!

[@elilavi7514]

Awesome as usual !

[@dataschool]

Thanks!

[@rishimusicprods]

This video is quite helpful and easy to understand. Thanks a lot!

[@dataschool]

You're welcome!

[@tommonks2490]

Great explanation. This was a huge help. Thanks so much!

[@dataschool]

You're very welcome!

[@jieqi6341]

Thank you so much! you are amazing as always ! I really appreciate it ! Please don't stop making these videos !

[@dataschool]

Thank you!

[@LonglongFeng]

very nice tutorial, your style of teaching is awesome like an amazing opera singer

[@dataschool]

What a compliment, thanks! :)

[@rehmanullahkhan7389]

Thank you so much.Very useful. I have no wordings to appreciate you. I liked your way of teaching very much. You became my ideal in teaching.

[@dataschool]

You're very welcome! Thanks so much for your kind words!

[@nimesharya909]

Amazing, clear, precise and I got it working as well :)

[@dataschool]

Great to hear!

[@sapnasinha804]

Fantastic explanation , however at the end would be good to mention that there are more ways to fill with value_counts , eg. With the mean of all other values etc and not just merging null column with any other column. Cheers!

[@dataschool]

Thanks!

[@mdfaiz4583]

great tutor...great way of making us understand.... so easy and intuitive

[@dataschool]

Thanks!

[@azbas]

Thank you for simple and detailed explanation including the use of features.

[@dataschool]

You're very welcome!

[@康朵朵-d1r]

all the content in the video are presented clear!!! thanks very much!! we love you!!

[@dataschool]

And we love you! 😉

[@gouravkushwaha4488]

You are good. Your explanation really made it simple.

[@dataschool]

Thanks!

[@HasanSuper]

What a beautiful video and such great explanation. Beautiful. Keep it up

[@dataschool]

Thank you! 🙏

[@BrokenLightPole]

Great video and explanation as always!

[@dataschool]

Thanks!

[@saranyan4123]

Thank you for the clear and quick explanation. Very helpful !!

[@dataschool]

You're welcome!

[@causap]

Markham, make America Great Again...You're the Boss..

[@dataschool]

Ha! Thanks very much :)

[@RahulKumar-bh9hb]

Explanation techniques is great........want to thank you for sharing your knowledge......Grt videos

[@bagushari1886]

Could you please make a video on how to handling missing values in multiple sheets in pandas? Or any recommendation source that I can read about it? Thanks in advance

[@dataschool]

Thanks for your suggestion!

[@pegasoos]

I watched your first video, you are legend!

[@dataschool]

Ha! Thanks :)

[@paula805]

What inspires a down vote on any of these videos?? Always great content!

[@dataschool]

Thanks Paul! :)

[@Andrew6James]

I thought we said that no values were missing from the City or Shape reported columns. Why do we see rows dropped at @11:07?

[@dataschool]

Values are missing from both the City and Shape Reported columns.

[@usmanshaikh1115]

Very useful and easily explained.

[@dataschool]

Thanks!

[@joehopewell]

Thank you!!!! All the good stuff, all in the same place...love it!

[@dataschool]

Great! :)

[@nataliaagudelo8635]

As always, your videos are very helpful!

[@dataschool]

Thanks very much for your kind words!

[@MrJioYoung]

Thank you! Great instructions!

[@dataschool]

You're welcome!

[@uttamkumarpatra7616]

You are simply awesome :) .thank you for making such wonderful videos

[@dataschool]

That's so nice of you to say - thank you!

[@nataliyakunderevych1211]

Super. I understood everything. Nice explanation

[@dataschool]

Thanks!

[@akashjoshi6826]

Fantastic video Sir.Your work is really commendable.It would be great if you can make a video about imputing the missing values in Python.

[@dataschool]

Thanks for your suggestion, and your kind comments!

[@alexhenning7086]

Superb video! Thanks a lot it helps alot !

[@dataschool]

Glad it helped!

[@ExcelTutorials1]

This is super helpful, thank you!!!!!

[@dataschool]

Glad it was helpful!

[@saubhagya594]

Lots of thanks from NEPAL✌✌✌

[@dataschool]

Most welcome 😊

[@nyashagracenhandara7757]

thank you so much the explanation is very clear

[@dataschool]

Glad it was helpful!

[@rajpaul1501]

Truly amazing videos. Can you do a series on Matplotlib and Seaborn

[@dataschool]

Thanks for your kind words and suggestion!

[@tresortshimbombo3133]

That's exactly what I was looking for!

[@dataschool]

Great to hear!

[@innocent.amchechesi276]

You are a saviour.... I'm happy

[@dataschool]

Thanks!

[@HJ-uy6ez]

Great video, how do I get the codes? Does any know or can help? Thanks

[@dataschool]

See here: nbviewer.jupyter.org/github/justmarkham/pandas-videos/blob/master/pandas.ipynb Hope that helps!

[@FE12343]

Amazing video, thanks!

[@DanielBRLima]

I just would like to thank you man.

[@dataschool]

😄

[@melvin9993]

I learned a lot through your videos!

[@dataschool]

Great to hear!

[@delmaregals]

Hi let's say I accidentally changed the value like the one I line 19 where NAN is change to Various can I reverse the change?

[@dataschool]

No, changes made through assignment (or inplace operations) are permanent!

[@oysteijo]

Hi Kevin! How can I fill na based on a condition? Say I want to fill NA for all missing cities, but only if the color is red.

[@dataschool]

Great question! ufo.loc[(ufo.City.isnull()) & (ufo['Colors Reported']=='RED'), 'City'] = 'New value'

[@ashishkhuraishy]

Man thx btw😁

[@joescanlon7502]

Excellent video series

[@dataschool]

Thank you!

[@jojithekkan3713]

I have a question. If I find value_counts, how can I select one or more records based on index names and/ or columns?. Thanks in advance

[@rohitsinghal2972]

Sir what you told in the video is applicable only for the numbers and what should be done for the string values?

[@dataschool]

If you like, you can impute missing string values with the most common values using scikit-learn's SimpleImputer.

[@SkillTop]

ROHIT SINGHAL Hi programmer🔌🤩 pleaaaase see my channel🌹

[@tyl9680]

In the last part of the video, why the number of "VARIOUS" made by fillna doesn't match the previous NA number?

[@traveltripfamily7486]

Very didactic! Congratulations!

[@dataschool]

Thanks!

[@simbaphiri3205]

Excellent video!

[@dataschool]

Thank you very much!

[@yashugarg1815]

Doubt: Sir , If I want to assign Na to a value suppose 5.Means where ever 5 is present in a DataFrame it will be replaced by Na.then how I have to proceed???? Thanks

[@mountainscott5274]

df.column_name.replace(5, np.nan, inplace = True) check to make sure values are replaced with df.info()

[@dataschool]

Nice!

[@NR_Tutorials]

thanks for Nice lecture we love ur sir

[@dataschool]

You're welcome!

[@jonathanfriz4410]

Hi, how you can handle the ValueError: arrays must all be same length ? when df.transpone() is not an option?

[@ekehopenkiruka]

Awesome, simple and straight to point with code, what i have been looking for weeks. Thank you so much. Do you have any video where you have used NSL KDD or KDD 99 data set to demonstrate data pre-processing as this is driving me naught.

[@dataschool]

I'm sorry, I don't have a video like that... good luck!

[@amitvajpeyee3890]

Thank you so much for explaining in such a simple language! You are doing a great job. God bless you!

[@dataschool]

You are so welcome!

[@nasser_omar]

What about displaying the rows where columns 'A' and 'B' both of them have any missing values?

[@eshaal2525]

Hi when I change na to nan in my data frame ...all I refers become floats... N first there was no bulk values showed ..but now there are null values

[@wesleypgurira7142]

hey , how can we replace a NaN value with the previous value in a database like on ufos (shapes ) instead of various you place maybe rectangle shape if it was before the NaN value

[@RelGuzman]

nice tuts dude

[@dataschool]

Thanks!

[@keshavkashyap2012]

In this Tutorial for finding the missing city name we used syntax ufo[ufo.City.isnull()] but what if i have to find the missing "Shape Reported" the syntax ufo[ufo.Shape Reported.isnull()] is not working? how to specify the space?

[@romulotrucco8282]

super useful!! thanks

[@dataschool]

You're welcome!

[@couthelle229]

I love your videos, thanks so much!

[@dataschool]

You're welcome! Thanks for your kind words :)

[@mingqian813]

Thanks for all your well-made videos! I got to know you and your classes from Datacamp. As a beginner in the ML field, please allow me to ask a silly question. So if we have categorical features with missing values, do we need to handle missing values first then do categorical feature transformation using encoders? Or the order doesn't matter? Thanks!

[@dataschool]

Great question! Previous to scikit-learn 0.24, missing values need to be handled first if you are going to one-hot encode them. Starting in 0.24, OneHotEncoder can handle missing values itself. Hope that helps!

[@kennethstephani692]

Great video!!

[@dataschool]

Thanks!

[@amishbhat3560]

You told how to handle NaN values but if there are some other values such as "Not Provided" then what to do? How to ignore them?

[@dataschool]

Excellent answer! 👏

[@leilimalek-k2s]

Thank you !! your videos are awesome

[@dataschool]

Thanks very much for your kind words!

[@soumiahm7318]

Thank you for the excellent videos

[@dataschool]

You're welcome!