How to Do Data Cleaning (step-by-step tutorial on real-life dataset)

[@LightHouse31073]

I would highly recommend new data analysists to participate this this data cleaning process you present. The level of detail is impeccable 👌

[@misraturp]

Great to hear Loyiso, thank you!

[@LightHouse31073]

@@misraturp My pleasure. You're so good. (*meant to say Data Analysts and not the gibberish above🤭)

[@ExtraKanin]

As someone with short attention span these days, I'd like to pat myself at the back for being able to sit through the entirety of the video. Data is really interesting! Thanks, Misra :)

[@misraturp]

Great to hear that you enjoyed it! :)

[@cybiryan]

Thank you for showing us these techniques. And I love the pace, thank you!

[@simenandreasknudsen9272]

This is great Misra, thanks! :) Please make more of these!

[@misraturp]

Thank you!

[@fehmidatahir2509]

Thank you SO much! I am totally in love with the content you share!

[@misraturp]

You're very welcome :)

[@antonioarana8002]

PERFECTION all this explanation and process, thanks Misra! you are a master, thanks a lot!

[@misraturp]

You're very welcome :) and thank you for your nice words Antonio :)

[@iqraasif3783]

This is gold. Learned a lot from your videos. Thanks!

[@prekshagampa5889]

Hey Misra! I am new here and I love your content. And, Thank you for this wonderful explanation, Its really helpful to get a clear view of how things work .

[@misraturp]

That’s great to hear. Thank you!

[@fernandoraposo5038]

That was the video that i was searching for. Thanks :)

[@misraturp]

Awesome! You are very welcome.

[@aaratigeorge2904]

This was a wonderful tutorial wherein you also gave insights on what things u consider while data cleaning, thank you.

[@misraturp]

You're very welcome!

[@ahmedkaram-ws3hs]

thank you misra for this great content please keep making videos it's very very helpful and your explanation is so much great.

[@misraturp]

Thank you Ahmed!

[@atharvasavdekar7613]

You deserve more appreciation. Great Content!!!

[@misraturp]

Thank you so much 😀

[@animelover5093]

Great content! This step-by-step guide has been extremely helpful in my journey.

[@KonradTamas]

Smart bute :) and great content ! Thanks

[@la-vieborde9825]

thank you so much. I am a new data scientist and these videos are very helpful.

[@misraturp]

Great to hear!

[@mohammedansar8023]

Amazing...My search has ended here... pls continue unloading more videos...

[@misraturp]

That's awesome to hear! Thank you.

[@KhaliDALKhafaji]

that was a very great illustration thanks alot

[@misraturp]

You're very welcome!

[@abdelrhmanrhyaseen6194]

This was really amazing, Thank you.

[@deandu3414]

Thank you for the amazing lecture.

[@raminsadeghnasab9310]

That was amazing. Thanks for your time.

[@misraturp]

You're very welcome :)

[@FRANKWHITE1996]

Thanks for sharing! 👍

[@misraturp]

Of course!

[@BillusTinnus]

So this is what data scientists do... Nice video ! :)

[@misraturp]

Yes, thanks!

[@judyostroot8682]

Wonderful tutorial! This is exactly the type of content I was looking for. When you are cleaning data in a work environment, do you document all the changes you make?

[@misraturp]

Hey Judy, I’m happy to hear that! I did not document the cleaning steps most of the time. But having clear comments on the code itself is very useful to people who will maintain the code after you to understand why you have done something.

[@paragandozdroch3791]

Thank you Misra for the detail video, very helpful. I am wandering what is the shortcut for the search drop down bar on 29:18 min of your video . Thanks

[@asadghnaim2332]

Please more on data cleaning

[@misraturp]

Noted!

[@rwejolandacademy]

I loved you from the time I see you. TYJ

[@8CountLife]

I'm very appreciative for your channel

[@misraturp]

Thank you 8CountLife!

[@hassanmahamat-pz8fx]

Good explanation.

[@onyinyeobijiofor7075]

This is so nice and clean 👌

[@misraturp]

Thank you! Cheers!

[@harikishan437]

It's really amazing, i learned a lot of new coding lines here as well ass the concept too ......Thanks a lot @Misra Turp😇😇😇😇😇

[@misraturp]

That's great to hear, thank you!

[@CresentX]

Good work Misra

[@misraturp]

Thank you :)

[@NotFog1]

So good, thanks a lot!

[@misraturp]

You're very welcome!

[@haliltezel8106]

Thank you for content mısra,datas not always clean as in tutorial:) Good job keep going my friend

[@misraturp]

Thank you!

[@okotpascal]

The playlist doesn't seem to be arranged for one to easily follow up on the videos. otherwise great content and thank you for the tutorials.

[@alifiaz7792]

Very well explained. During the cleaning you arbitrarily took 25th and 75th percentile as limits for cleaning tree diameter. Can you recommend a more systematic approach to select the these lower and upper quantile values? So appropriate treatment can be applied to the values below and above.

[@misraturp]

Hey Ali, great question. My main goal was to not drag the video for very long so I didn't go very deep into that decision. You can read of a good description on this page (machinelearningmastery.com/how-to-use-statistics-to-identify-outliers-in-data/) under the "Interquartile Range Method".

[@alifiaz7792]

​@@misraturp Thanks Misra for sharing the link

[@nicolaslpf]

Most people use IQR

[@diegomartins7214]

Thank you!

[@ecitahpi385]

Dear Misra, thank you for your tutorial. from 19:40 min with merge codes and other till and other codes unfortunately not working by myside.. is it possible to share your code source in GitHub, too?

[@searchbug]

Wow! Not everyone gets to share this kind of in-depth walk through. Thanks for sharing this, Misra! For our dear friends who are not really techy or no experience dealing with codes and stuff, you may learn the basics or find others ways to clean data. Bulk validation, for example, the only thing you need to do is to make sure that your data sets are well-organized, with proper headings. You can then simply upload them in a third-party software that will do the verification for you, which then will let you know what data sets are outdated, invalid, and inactive.

[@misraturp]

You are welcome! Thank you for the addition.

[@searchbug]

@@misraturp My pleasure! Thank you for considering it :)

[@idongessien2245]

Not only are you pretty but freaking intelligent. (Forgive my choice of words, but I'm always straight forward). Was almost frustrated at some point...Thank God I ran into your channel

[@kunjalsahu3504]

Great content mam for fresher keep up the good work , new subbie

[@misraturp]

Welcome!

[@hamidmirza333]

Grateful for awesome tutorials. How to split the datasets for training and testing sets? I am working with EMG signal classification using SVM classifier. I am confused how to split the data sets to do classificaiton task.

[@asadghnaim2332]

Thank you a lot Mirsa :D

[@misraturp]

You're welcome 😊

[@dominicatuahene7303]

Hi Misra, please the link to the previous video on before this one

[@misraturp]

Here it is: kzbin.info/www/bejne/hYqXloSomtCErNU

[@HasanKarakus]

Harika anlatım !!

[@misraturp]

Tesekkurler!

[@shivamagarwal587]

mask=((tree_census_subset['status']=="Stump") | (tree_census_subset['status']=="Dead")) this line given an error to me can you explain what's the problem here

[@govindant8360]

Very useful for me Mam.

[@misraturp]

Thanks a lot

[@rangabharath4253]

Awesome 👍😎

[@misraturp]

Thanks ✌️

[@ashikinfodu]

Hi Misra, a quick question, what you do with missing values created by logical/skip questions in the survey? Thanks?

[@md.alamintalukder3261]

Great ❤

[@tejaswaroopdasari1495]

Your look Great by subject wise Content

[@misraturp]

Thank you.

[@prasadxev]

when angel teaching programming

[@shameermasroor7375]

Hello, Misra! great video! I had a question. When I run the tree_census_subset['steward'].value_counts() statement, pandas does not return the None count, or the count of the trees which do not have a steward. Has there been some change in the value_counts() function or am i doing something wrong?

[@shuklajitechnicals2907]

hey, what font( code font ) you are using ??

[@ankitsrivastava06]

Please provide datasets for practice csv file

[@mkhex87]

isn't it often useful to leave missing values as np.NaN so that your aggregation functions skip over them but still compute?

[@misraturp]

Depending on your goal that would be helpful. When preparing data for a machine learning task, we would like to get rid of or correct as many missing values as possible so they don't crush the training process.

[@samirmendhe7387]

You made this so beutifull

[@misraturp]

Thank you!

[@hilloldasfisheuphoria]

Hallo Misra, while enrolling myself to the course "deep learning 101 with python and keras" a "coupon code" is asked, how do I get it ???? where from I get the coupon code so that I can fill the "Add Coupon Code" field ???? ,....

[@misraturp]

Hey Hillol, that field is optional. I occasionally have campaigns, then I create a coupon code. There are no eligible coupons right now.

[@vinayakdixit2855]

@19:40

[@minhhapham3010]

you are so beautiful, and the the content of video is very an useful for new learner. Many thanks

[@ekaterinakorneeva4792]

Please make the links stay on the screen for more than 1 second, it would be much more convenient. Thank you.

[@MOHANAMona-yq3dl]

Where can i get the data set?

[@misraturp]

I believe this is the dataset I'm using. www.nyc.gov/site/tlc/about/tlc-trip-record-data.page

[@MOHANAMona-yq3dl]

@@misraturp Thank you ✨

[@DataSet]

Im ready to be cleaned

[@um1541]

Would it be better to find 75% and 25% from the max value, instead of using percentiles? 15 doesn't look like 75% of 59, so I assume, we are editing 50% of data.

[@mad1337nes]

that's the quartile markers, not an actual percentage. The mean of the "bottom 25% of the data" is that number, same as the mean of "the top 75%", not 75% of the max value

[@parkuuu]

Hello Misra, For the last part (substituted the diameter with Q1 and Q3 values), wouldn't it be better to retain the original column data and then just add a new one based on a condition (like np.where actual dia < Q1, use Q1 else dia), just to be able to compare it side by side

[@مسافر-ح2ط]

I thought cleaning should be before exploration?

[@dr.emmrich]

How do you clean a data you haven't explored?.. the exploration actually reveals what to clean

[@kiwi-mf2do]

Can't this get automated with Genrative AI?

[@antukhan5592]

can u share github code?

[@hilloldasfisheuphoria]

Misra I had submitted my "first name" and "email" to get "Pandas Cheat Sheet (free)" three times, but I have not received any mail yet !!!! ,....

[@misraturp]

Hey Hillol, could you check your spam folder? It looks like the email was sent.

[@keidran_r3]

data wrangler, a vs code extension. you're welcome.

[@ttffan658]

Cute smile

[@salimayad2151]

She can fix me

[@rohitbuddabathina]

Did anyone tell you that you resemble Angelina Jolie ?😀

[@misraturp]

Not until now. I'm flattered. 😅

[@rohitbuddabathina]

@@misraturp I hope you run a correlation test on your face and Angelina's face. I really wanna see the results 😀

[@misraturp]

@@rohitbuddabathina Haha sure. :D

[@msumode4493]

Visiting for your face.

[@theamithsingh]

Great video series @misraturp

[@misraturp]

Thank you!

[@misraturp]

👉 Get real world data science experience by doing hands-on work www.misraturp.com/hods

[@namaniitkanpur5697]

you looks very beautiful