Real World Data Cleaning in Python Pandas (Step By Step)

In this video, I show you how to clean up data within Python Pandas within Jupyter notebook. This Python tutorial is great for those trying to get into Data Analytics or Data Science.
Cricket Data: www.espncricinfo.com/records/...
Everything is coded within MSSQL and inside SQL Server Management Studio.
Interested in discussing a Data or AI project? Feel free to reach out via email or simply complete the contact form on my website.
📧 Email: [e-mail]
🌐 Website & Blog: ryannolandata.com/
🍿 WATCH NEXT
Python for Data Analyst and Scientists Playlist: • Python Tutorials
Python Groupby: • The Complete Guide to ...
Python Pandas Interview Questions: • 23 Python Pandas Codin...
Python Lambda Functions: • Python Pandas Lambda F...
MY OTHER SOCIALS:
👨‍💻 LinkedIn: / ryan-p-nolan
🐦 Twitter: / ryannolan_
⚙️ GitHub: github.com/RyanNolanData
🖥️ Discord: / discord
📚 *Practice SQL & Python Interview Questions: stratascratch.com/?via=ryan
WHO AM I?
As a full-time data analyst/scientist at a fintech company specializing in combating fraud within underwriting and risk, I've transitioned from my background in Electrical Engineering to pursue my true passion: data. In this dynamic field, I've discovered a profound interest in leveraging data analytics to address complex challenges in the financial sector.
This KZbin channel serves as both a platform for sharing knowledge and a personal journey of continuous learning. With a commitment to growth, I aim to expand my skill set by publishing 2 to 3 new videos each week, delving into various aspects of data analytics/science and Artificial Intelligence. Join me on this exciting journey as we explore the endless possibilities of data together.
*This is an affiliate program. I may receive a small portion of the final sale at no extra cost to you.

[@ArmanKHAN-bj9iv]

Fantastic tutorial! Your step-by-step guide on data cleaning in Python Pandas was excellent. Clear explanations and practical examples made it easy to follow along. Looking forward to more of your uploads. Keep up the great work!

[@RyanNolanData]

Thank you! I’ll have another Python video up this week as well as more coming soon!

[@AJAY7509]

this video really helped me man, i was trying to leard about panda now it poped up on my notification, thanks for the video.

[@RyanNolanData]

No problem check out my other pandas vids I have a full playlist

[@nickdaboss03]

you work super hard and put out really good content. Keep it up man, I'm looking forward to watching you grow!

[@RyanNolanData]

Thank you! Have another video ready to go later this week as well as 90% done with another Python interview question video.

[@Al-Ahdal]

@Ryan Nolan: Excellent Video. Very clearly explained. I'm looking forward to watching you grow!

[@RyanNolanData]

Much appreciated!

[@koo5867]

Now that’s some cool content. This is exact what I wanted. Thanks bro🙏🏼keep helping the poor students like us! 😌

[@RyanNolanData]

No problem

[@nagamanickam6604]

Thank you Ryan nolan

[@RyanNolanData]

no problem

[@tapspasi2319]

Amazing! Very good presentation

[@RyanNolanData]

Thank you

[@loydteds3944]

You're video is very helpful! One question though, how do you remove duplicates in high dimensional data, lets say with 500 duplicates? Thanks

[@user-iu5nz2gy6l]

Thanks . Appreciate for this tutorial. Just have a question on Q5. Why is it already in a data frame? while we have to use to_frame for Q4 ? Thanks

[@yankoshuan6225]

i guess watching your videos while preparing my own portofolio , i am halfway there. Thanks a lot

[@RyanNolanData]

No problem. My first batch of classification vids are done working on regression now

[@prathmesh_jadhav8930]

Brother you doing awesome…. Upload more videos related to data analysis

[@RyanNolanData]

I have a full playlist of 70ish vids! Working on more though

[@lewismurigi3623]

This was so much helpfull, Thanks Man

[@RyanNolanData]

No problem

[@RyanNolanData]

No problem

[@tianbowen721]

Pretty Amazing :) and I'd say it's some dense content to fit in 40 mins ~~I learned a lot

[@RyanNolanData]

Awesome

[@far3582]

I am trying to move away from R, and this is a great video. Thanks Ryan!

[@RyanNolanData]

No problem best of luck

[@SuccessGossips]

star means not out with highest score, you don't need to remove it

[@ArhamZaiem]

In the highest inns score, why didn't you used rstrip to remove * instead of split??

[@satishharijan7280]

nice lecture bro thanks for this it is use full video for me

[@RyanNolanData]

No problem

[@yvonnemukhono3566]

Very helpful.

[@RyanNolanData]

No problem

[@pradeeppadeliya]

This is a best tutorial .... 👍👍👍👍👍👍👍👍👍👍👍👍👍👍

[@RyanNolanData]

Means a ton thank you

[@Al-Ahdal]

@Ryan Nolan: Your videos are great indeed. It is requested to have a comprehensive series on "Data Analytics & Visualization". Thanks

[@RyanNolanData]

I have a full data Analyst playlist check it out

[@Al-Ahdal]

@@RyanNolanData , could you please tag or locate. Thanks

[@RyanNolanData]

@@Al-Ahdal kzbin.info/aero/PLcQVY5V2UY4JrrKi2bW7DdOD08shTs4QQ

[@benayawilly6536]

good work. keep it up

[@RyanNolanData]

Thank you! I just uploaded a new video

[@user-bf9lq6bb5s]

Totally it was a great effort and much appreciated for your hard work. I would like to know how to remove or drop null values from the columns. Thanks in advance

[@RyanNolanData]

Look up drop na

[@user-bf9lq6bb5s]

Cheers man... any advice how to remove year from a columns. for instances, if a column has numeric and year values and want to remove year (2004 in format)only.@@RyanNolanData

[@khan07700]

Sir when we import data from site to table I'm not getting the option of table 0 what's the solution for that at 1:54.

[@salfrat55]

"FS Jackson played for Cambridge University, Yorkshire and England. He spotted the talent of Ranjitsinhji when the latter, owing to his unorthodox batting and his race, was struggling to find a place for himself in the university side, and as captain was responsible for Ranji's inclusion in the Cambridge First XI and the awarding of his Blue. According to Alan Gibson this was "a much more controversial thing to do than would seem possible to us now". He was named a Wisden Cricketer of the Year in 1894. He captained England in five Test matches in 1905, winning two and drawing three to retain The Ashes. Captaining England for the first time, he won all five tosses and topped the batting and bowling averages for both sides, with 492 runs at 70.28 and 13 wickets at 15.46. These were the last of his 20 Test matches, all played at home as he could not spare the time to tour."

[@RyanNolanData]

Didn’t know this is a really cool story. Like Branch Rickey in baseball

[@pavankalyan_297]

The star in the Highest score column means they were not out till the end of the match. Great tutorial Ryan. will it be possible for you to attach the notebook file here

[@RyanNolanData]

Thank you and I can look at adding the code to Github this weekend

[@user-vl3hm9hv3x]

bro...u should have used replace method with regex for cleaning *,+ etc chars from the columns

[@RyanNolanData]

I used regex in my latest project and have a video coming out on it soon funny enough

[@davideschreiber2821]

Lots of good stuff here, but I finally gave up at 31:24. If you're confused about what's happening, imagine how confused we learners are as you bounce around from cell to cell copying-pasting-deleting-trying again, trying to figure things out.

[@RyanNolanData]

Bugs are part of programming and no one is perfect. I show how it’s solved and why it happens

[@MrFravallec]

Great tutorial, got this issue on the data types: AttributeError Traceback (most recent call last) Cell In[11], line 1 ----> 1 df['Inns']= df["Inns"].str.split(pat = '*').str[0] File ~\anaconda3\Lib\site-packages\pandas\core\generic.py:5902, in NDFrame.__getattr__(self, name) 5895 if ( 5896 name not in self._internal_names_set 5897 and name not in self._metadata 5898 and name not in self._accessors 5899 and self._info_axis._can_hold_identifiers_and_holds_name(name) 5900 ): 5901 return self[name] -> 5902 return object.__getattribute__(self, name) File ~\anaconda3\Lib\site-packages\pandas\core\accessor.py:182, in CachedAccessor.__get__(self, obj, cls) 179 if obj is None: 180 # we're accessing the attribute of the class, i.e., Dataset.geo 181 return self._accessor --> 182 accessor_obj = self._accessor(obj) 183 # Replace the property with the accessor object. Inspired by: 184 # www.pydanny.com/cached-property.html 185 # We need to use object.__setattr__ because we overwrite __setattr__ on 186 # NDFrame 187 object.__setattr__(obj, self._name, accessor_obj) File ~\anaconda3\Lib\site-packages\pandas\core\strings\accessor.py:181, in StringMethods.__init__(self, data) 178 def __init__(self, data) -> None: 179 from pandas.core.arrays.string_ import StringDtype --> 181 self._inferred_dtype = self._validate(data) 182 self._is_categorical = is_categorical_dtype(data.dtype) 183 self._is_string = isinstance(data.dtype, StringDtype) File ~\anaconda3\Lib\site-packages\pandas\core\strings\accessor.py:235, in StringMethods._validate(data) 232 inferred_dtype = lib.infer_dtype(values, skipna=True) 234 if inferred_dtype not in allowed_types: --> 235 raise AttributeError("Can only use .str accessor with string values!") 236 return inferred_dtype AttributeError: Can only use .str accessor with string values!

[@Muhammad.Kashif31]

your data may be containing integer data, thats why you are getting the error

[@VladislavShishkin11]

I completed the project but I reopped it today and all the code was still there, but when I typed df it was the old table uncleaned? how do I make sure this doesn't happen again?

[@RyanNolanData]

Ill add my code to github this weekend

[@marcus.the.younger]

save the cleaned data

[@hemantsharma-xf3ub]

where i can get the notes

[@tasmisa6778]

How am I supposed to know all the alphabets are named as those you just did???

[@taha5754]

Can you share the notebook used in this tutorial? @RyanNolanData

[@RyanNolanData]

I need to make a website article on this. It’ll have the code in there

[@sachinnambiar]

Its a dictionary right? Not a list. #rename multiple columns in a dictionary

[@dogzrgood]

Star * means the batsman was not out 😊

[@RyanNolanData]

I appreciate it. Didn’t know

[@Al-Ahdal]

@@RyanNolanData , Yes * mean batsman not out, but it won't affect any calculations. Great work indeed.

[@rajareddyraju6773]

19:09

[@salfrat55]

Headley @4 min mark 😂😁

[@RyanNolanData]

Haha one day I’ll buy your dup