Statistics-Finding Outliers in Dataset using Z- score and IQR

[@satyaprakash5905]

Z-Score & Outliers - explained in very easy way. Thank you..

[@prathameshjoshi1408]

Thank You so much for explaining most of the concepts in simplest way.When you explain anything by far anyone can say you really want us ( your students ) to learn more . One of the best teachers for sure. Thankyou and best of luck for further success on KZbin.

[@shivambhayre5056]

Thanx bro mujhe ye sb pta tha i mean pure theory and formulas but implement kese krte h python me ye tumne bahut easy way me explain kia thanx man really appreciate that and me wait krunga agle video ka jisme real data p kre thanx for that because me R p ye kr leta hu but i am not good in python aur tum bahut sahi tarike se smjhate ho thnx for that🙏

[@safwanmansuri792]

You continue this hardwork ,surely you will get success one day on KZbin.

[@ayushijmusic]

Best!!! Thankyou! This channel is a saviour in DS doubts

[@vsabinat]

Wonderfully explained. Won't forget now. Thank you

[@justfun6409]

Could you please arrange the playlist videos in 1st to last manner? Just I am getting confused that which video I should watch after particular video. By the way, you explain very well👍🏻💥

[@pawansonu41]

at 14:13 point 4 and point 5 are wrong..however in code you have implemented correctly. A suggestion please "try to keep formulas and notes clean, because if that is ruined whole plot gets off". Though you have done a good job by far.

[@sandipansarkar9211]

Awesome explanation Krish.Thanks

[@shashankverma4044]

Thank you so much ...Concept cleared perfectly:)

[@sudiptachakraborty745]

Sir - I appreciate your effort of making all these beautiful videos. It makes life easy for all of us. But I need to make one comment on this video, at the beginning you said when the weight(x) is increasing height(y) is also increasing but in real world if you think it's not possible right ? As per my thought process may be you wanted to say when the height(x) is increasing weight(y) is also in a increasing trend. In most of the cases, when a person's height increases automatically weight also get increased but it's not the reverse. I hope i am making sense here. Thank you once again :)

[@mannudhapola1210]

Smoothly explained

[@idanamayank]

Simply awesome,great work

[@balajivarma83]

Sir please upload you kaggle competition of house price with feature selection. Waiting for your vedio

[@harshtamkiya8505]

Nice explanation..

[@ty_b_63_prajwalwaykos86]

Hi Krish, its always great to learn from you... can anyone please tell when to use what? Or which amonthe the methds is best? or is it case wise and if yes what are the cases?

[@rambaldotra2221]

Thanks a Lot Sir, Really helpful.

[@sheelstera]

Standard Normal distribution can be applied only to a normally distributed column...SND applied to a non-normally distributed column may give you a mean of 0 and and SD = 1 but the nature of distribution is still non-normal. Z score cannot be directly tied to the 99.7% of ND in such a case. The interpretation of Z score is therefore subject to the kind of the distribution structure of the column which remains unaltered before and after the Standardization process.

[@yendamurikrishnavamsi3188]

super!!! well explained sir!!!!!

[@shivambhayre5056]

If we have many outliers, in that case, is it good to calculate(or detect by frequency)mode over mean and median ?? And then replace them with mode (mostly we do it for categorical variables) is this a good way?????

[@louerleseigneur4532]

Thanks Krish

[@janardanpandey6942]

Here outliers is exactly coming in Z score when I'm putting the value of threshold=1 and not coming when putting the value of threshold=3 WHY????? Btw thaxx alot very outstanding playlist for STATISTICS.....Gtr work

[@rahulgarg6363]

perfectly explained krish but How to detect outliers for multidimensional data

[@TheTimtimtimtam]

Nice

[@umesh789s]

This was really explained very well. I have a doubt regarding multipication by 1.5 , can you please elaborate how this 1.5 came or what is the method by which we can get this value in any other dataset.

[@umesh789s]

Now i can breifly explain why only 1.5 and the reason is really amazing and really informative.

[@KumarGolu2001]

Can you explain me ?

[@lokeshkaturi4040]

@@KumarGolu2001 It is just a proof done by stats people , 1.5 gave the right bound values to detect outliers.

[@baharehghanbarikondori1965]

amazing tutorial, thank you

[@vicky-do5th]

sir why you have taken threshold as 3 as you said it should fall in 3rd standard deviation ,so it should not be (3*std)?

[@umesh789s]

I am not able to find the link of the next video which you have told related to outlier handling in a dataset from kaggle.

[@KumarGolu2001]

I also not getting the video

[@akshatabm4491]

Is the detect_outlier using z score program working fine for everyone?? It is not returning correct output for a diff dataset used.

[@Daily_Dose_Of_Life]

Thank you, sir Finding Outliers with help of Z Score Using Pandas: Just some modifications outliers=[] def detect_outliers(df): threshold=3 # 3 standard deviations # mean=np.mean(df) #std=np.std(df) z_score=(df-df.mean())/df.std() df['out']=z_score[z_score>threshold] df.dropna(inplace=True) return df.iloc[:,0].values.tolist() detect_outliers(d)

[@AkshaySharma-vm9sp]

Sir please make video on p value

[@rohitsharma-kr9gk]

Sir can you make a video on fitness based recommendation system. and can you tell me how to collect data set of different people and recommend them food and exercise according to that data set. plz reply sir....

[@samitaadhikari3182]

actually that is very good idea i hope you learned that i'm gonna try this once i learned this

[@ukquaratine1019]

can the value of inter quantile would be range in negative(-)

[@abishekkachroo938]

I am facing a unique issue , following this methdology for IQR: Mine lower value of column is coming . Below is the pseudo Code: Q1 = df[' shares'].quantile(0.25) Q3 = df[' shares'].quantile(0.75) IQR = Q3 - Q1 low = Q1 - 1.5 * IQR ------>>>>>>>>> this value is coming extreme negative which is not even present in columns The problem maybe is IQR value > Q1. How to solve ? Shall I change the % age?

[@meenadalvi9743]

In the detect outlier function "y" is not defined anywhere please change it to i

[@prateshtamhankar3568]

we can use any one of the technique 1.Z-score or 2.IQR to find outliers??

[@niveditaparab6772]

generally when data is skewed IQR better if roughly normal then z score better

[@kadhirn4792]

Damn good dude. Thank you so much.

[@903vishnu]

If there are more number of features (like X1 X2 X3 and so on) then how can we find out using Scotter plot

[@cvb6931]

I have few doubts... why threshold value is considered as 3 I have seen other examples they also considering as 3 only ? is it always 3 ? why to only multiply with 1.5 for lower and upper bound values ... any specific reason? Does iQR works on categorical columns also ... I know that zscore doesn't work for categorical values?

[@ty_b_63_prajwalwaykos86]

Answering your doubts--- 1. it is not really 3 actually it is 3*sd where sd is 1(since by using z-score we make the normal distribution with mean=0 and sd =1) and yes, we always take the 3rd sd. that is all the values which are beyond 3rd sd are outliers. because until we reach 3rd sd 99.7% values are covered. which seems pretty enough. 2. As krish stated 1.5 is a statistical figure and drawn out of experiments so basically we have to use it. 3. It is very obvious ....using such statistics on categorical data doesn't make any sense. If you still not get it. just tell me how exactly are you going to sort categorical values at the first place😅😂😂🤣

[@ash_engineering]

Sir how is this method scalable when we have 50s or 100s of dimension /features in a dataset

[@anirudhyadav-mq8nd]

IQR method can be used when the data is huge

[@tamilanaroundtheworld]

Hi Krish, I have a doubt regarding the parameter inside the function , when defining the function ...it is given as data. It should be the dataset right ?

[@shantomatt]

data is just the parameter name , can be any label

[@akankshagupta5067]

In your code, after are np.abs(z_score) > threshold: Why are you appending y?? Shouldn't you append i?? Please correct it as it is misleading.

[@namansharma9697]

how can mean be equal to zero in real-time scenario as weight cannot be assigned a negative value please help

[@lifeisfun9]

Sir. Firstly a brilliant teaching and mentor. I have some doubt in calculating upper bound and lower bound you showed. You are saying anything beyond 1.5iqr will be outlier, then why we are doing low= q1-1.5iqr and high=q3+1.5iqr. cant we straight away get outliers by comapring if number is less than or greater than 1.5iqr. I did not understand meaning of q1-1.5iqr as lower bound and similarly for upper bound. How?

[@lifeisfun9]

Oh I understood . I read the text later. Thankyou so much for sharing your knowledge

[@Joshua75623]

Akansha, did you got the answer for your question? Even Im getting the same doubt

[@BehindTheLogics]

@@Joshua75623 First the values are sorted in increasing order. Q1 is the 25th percentile, Q3 is the 75th percentile. IQR is the length of the box. so values > Q3+1.5times IQR and values < Q1+1.5times IQR indicates the outlier.

[@ele_wings7521]

thanks alot... sir..

[@pranjalmittal7475]

sir please upload video on preprocessing data using sklearn library

[@knavk1]

Hi.. How to work with outliers.. I don't want to remove outliers from my data. Is there a way which I can do that?

[@anchitbhushan6172]

So which one should we use for removing the outliers??

[@lokeshkaturi4040]

Exploring all the ways is the best thing.

[@shahzan525]

i couldn't find the next video which you talk about in the last....

[@indirajithkv7793]

❤💫

[@srinathganesh6985]

can categorical data (encoded) have outliers?

[@lokeshkaturi4040]

Here is a nice explanation about how to detect outliers on categorical data .

[@dreamday4810]

@@lokeshkaturi4040 can you please share timestamp in this video

[@nayazithousifkhan1696]

Post the video of box plot technique

[@BehindTheLogics]

@nayazi kzbin.info/www/bejne/eYfHlKWVgqhlgbc Watch this video from 6.25

[@shantomatt]

plt.boxplot(x=dataset)

[@forammodi5850]

Sir i got error name error Nameerror: name mean is not defined

[@maithreshpalemkota]

It is more appropriate to call it Quartile than calling them Quantile.

[@sandipanpaul1994]

@krish why it is only 1.5 . Why not 1 or 2. Why only 1.5

[@amalsunil4722]

it's proven mathematically comparing with the standard normal distribution ....actually it's 1.7 but it's taken to be 1.5 for 'symmetry'

[@tammy4994]

how he has set threshold as 3?

[@TechBinod]

Your video is great but not systemically managed

[@rajprajapati888]

Z-score isn't give output for more than 3 outliers

[@ashwinimandani2829]

Sir, I tried finding the outliers in a dataset using both methods(Z-Score,IQR) and the answers are different. Can you please explain why?

[@amalsunil4722]

yea that happens....cuz mathematically proven: 1.7 is the accurate measure for the length of whiskers but it's taken as 1.5 for "symmetry". So you may check with 1.7 instead of 1.5 and observe the same result for z-score and iqr

[@jiteshmishra6949]

How to detect and remove from each column in one shot