Correcting Skewed Data with Scipy and Numpy

Skewed data can adversely affect your analysis and machine learning models. In this video, I demonstrate five methods for cleaning skewed data using the NumPy and SciPy modules. The methods include taking the square root, cube root, fourth root, log, and Yeo-Johnson transform. I also showcase the effectiveness of each method by summarizing the skewness of the data after each transformation with a bar plot.

[@marcom5873]

First time I have seen your videos. This is genuinely a very good video. Very well explained and clear. I am subscribing. The music wasn’t off putting either!

[@CJP3]

Thank you so much!!! I really appreciate it. If there’s anything you’d like to see just let me know!

[@marcom5873]

@@CJP3Sent you an invite on LinkedIn!

[@officialscience101]

the on-screen text is a great addition, Dr. P!

[@CJP3]

🙏🏽, I’ll incorporate more in upcoming videos! Thanks for the feedback!

[@metinunlu_]

Thank you for the video, subscribed! KZbin needs more quality content like this.

[@MikitaRashetnikau]

Amazing video I like it's structure: motivation, overview with examples, practical advices Thanks!

[@CJP3]

Thanks for the feedback! I’ll do more of this style!

[@mushinart]

Outstanding explanation, professor

[@CJP3]

Thank you so much!

[@dannybee9068]

Thank you! That was helpful! So we basically can make the root of any power? Is there a drawbag for exploiting it , like keep increasing the n value for feature to the power of 1/n?

[@CJP3]

Hi Danny! Context definitely matters. For analytical chemistry 1/n scaling is usually ok. a few downsides are that it makes the models less sensitive to potential outliers. Also its not suitable for certain distributions. Lastly, because 1/n scaling is non-linear, it can make data interpretation more difficult.

[@Lendemeier]

Bro this is data science ASMR 🤤

[@CJP3]

Hahaha I didn’t mean for it to be but glad you enjoyed it (I hope) 😂

[@pabloagogo1]

This is interesting. If one corrects the original skewed data, via doing these kinds of transformations, in the context of linear regression or multiple linear regression, will that not change the interpretation of the original data. Curious to know.

[@CJP3]

Perhaps, but that change may be for the better. I’d say it’s worth considering these transformation if you know you have skewed data. Many models especially linear models assume normally distributed variables. I usually build models with and without significant preprocessing and feature scaling/engineering.

[@nicolaslpf]

Amazing video! I was creating a function for measuring the same you forgot to name log1p Wich is log of (x+1) really useful for right skewed data with values less than 1

[@AyahuascaDataScientist]

Skewing doesn’t necessarily matter if you’re using XGBoost, correct? For classification or regression, that is

[@CJP3]

Exactly! Skewed data doesn’t impact all model frameworks.

[@thoniasenna2330]

SUBSCRIBED! What should one do before? Or, what's the correct order? - treating outliers, impute missing values, correct symmetry? Thanks Dr. P!

[@CJP3]

You’re not going to like the answer 😂… it depends a lot on the application. It’s first best to be aware they exist and then evaluate their impact on your outcome. For example if you’re trying to determine outlier samples - then outlier msmts wouldn’t be so bad.. maybe. Or missing values could be useful depending on the application so instead of imputing maybe you engineer a new feature.

[@CJP3]

Don’t unsubscribe after my answer! 😂 🤣

[@Stardust_Byproduct]

So what about if we were to standardize using z-scoring? It seems like that would get largely the same impact, wouldn't it?

[@CJP3]

Howdy, Z-scaling won’t improve the skew. The data will be mean-centered but will carry the non-uniform distribution

[@Stardust_Byproduct]

@@CJP3 that explains it. Thanks!

[@CJP3]

@@Stardust_Byproduct I think I’ll make a video that graphically illustrates this point. Thanks for asking :)

[@Stardust_Byproduct]

@@CJP3 yes that would be amazing! Thank you!

[@noobmasteroo69]

You're doing great, please avoid bg music while explaining. Thank you.

[@CJP3]

Thanks for the feedback! Glad you enjoyed the video

[@pewkaboo]

What if my data contains a lot of useful '0' values?

[@CJP3]

Howdy! Can you explain more about the 0’s?

[@pewkaboo]

@@CJP3 it is a expenditure data where the budget column contains a lot of '0' (not null) values.

[@prathambhatnagar8653]

please dont add background music

[@CJP3]

Thanks for the feedback. Most of the newer coding tutorials don’t have background music. Have a great day!

[@AyahuascaDataScientist]

I like it. Don’t listen to this hater!

[@mouhsineelqesry9446]

Bro you explain a concept, but go you need the music!! It’s distracting

[@CJP3]

I 💯 understand, they newer videos don’t have the music and the audio has a better EQ :)