Removing Outliers From a Dataset

[@theg.ksinstinct9344]

Great video! I did not know before how to get z-scores with this easy way, I always computed it using the formula. More appropriate way to deal with outliers, in case when you don't want to keep them and just remove them, then get the z-score for your variable of interest just as the lady did. Check the z-score and the variable together by both ascending and descending order of the z-score if you want, just as the lady did. Then had to the data section, select cases, select if condition is satisfied, write the formula in the formula portion - ABS(z_price)

[@JustKevStockholm]

Boxplots are also very useful - from a visual standpoint - when checking for outliers :)

[@WInspire]

you should do a formal test for outliers, not visual inspection of boxplot

[@menzir]

Do you have any references I can use to cite this method in my thesis?

[@rsx22Z]

How can i determine the critical value which helps delimitate the outliers. For instance your value was 3.29, but why >?

[@iwillsearchtheuniverse5371]

yess im still confused abt this

[@RectalDesign]

God bless you. Every piece of documentation and every guide I've read tells me "just delete the crazy values lol."

[@myfictionalthoughts]

Thanks for the advice, but how do I know which ones are the actual outliers? My data is not as simple as this one in this video.

[@benotoole1990]

trying to do this for university and we have the same surname, the universe works in weird ways

[@martinbrestovansky7813]

thanks Siobhan, really useful video, but what if I have outliers on both sides of the histogram? How can I set up the filter in Variable View Missing Column in this case?

[@sharist9343]

Hey, thanks for the amazing video. Would it somehow possible to quote the techniques you used? really would like to use this in my thesis. Kind regards :)

[@Electrify85]

Why 3.29? Isn't 1.96 was the key number?

[@SiobhanPhD]

Benjamin Smith 1.96 and 3.29 are both very important numbers in stats. 95% of "normal" data will be within 1.96 standard deviations of the mean. We use 1.96 for lot's of things like 95% confidence intervals, alphas of .05... For outliers however, 1.96 would be far too stringent - we'd eliminate data that really isn't outliers - data points that really do belong in our data set. So to make sure we are only excluding data that really is extreme we use 3.29. 99.9% of "normally distributed" data will be within 3.29 standard deviations of the mean. So if you have a data point that has a z value of 3.6 let's say, it is an extremely unlikely data point. Granted it isn't an impossible one - but it is highly unlikely and therefore considered an outlier.

[@chanothebest]

+Siobhan O'Toole using 3.29 for my data , mine is still not normally distributed. So, would it be right to use 1.96 to make it more normally distributed.

[@angelperdomo5]

+Jamie Frederick Howard look at z distribution table. These are standardized scores. 0 lies in the middle (mean)

[@mattadoritmo]

Holy crap! Thank you so much for this! This just helped solve a problem I've been working on for a couple days

[@Brickkzz]

Is there an automated way of removing outliers from positive and negative end? I have a data set of 120 000 cases and hundreds of outliers...

[@Brickkzz]

+Bob. Okay, I found a solution :) Use DATA -> SELECT CASES... Then in the Select window, choose: If condition is satisfied: *click IF* then type the name of your Z- variable < 3.29 and name of your Z-variable > -3.29. e.g. ZVALUE < 3.29 and ZVALUE >-3.29. This way SPSS will only use the central 99.9% of your cases for analyses :)

[@zivleong5151]

if i delete it~~the previous test such as normality test and compute mean need to redo?or will change automatically?

[@ivantau9753]

Well I believe you first need to test the data for outliers and once you did, and remove the outliers, then you can test the data...

[@PurpleRawr2]

Great video. I don't understand the 'id' section though. Is this just the IV? What if I have two IV's? Thanks!

[@Mysticflamee]

ID is a subject number.

[@SiobhanPhD]

Gizem ateş Yes exactly, it's just a subject or respondent identification (ID) number. A lot of my students want to use the numbers that are listed in SPSS as ID numbers but that doesn't work. Once you sort the data those numbers no longer line up with the same cases/subjects and then you are lost. If you find a data entry error you have to go through sometimes hundreds of files to figure out which one it is and correct the data entry error. And everyone, EVERYONE, I've ever worked with has made at least a couple of data entry errors. You IVs will go in columns labeled with whatever names you want to give them somewhere to the right of the ID number. Hope that helps.

[@ScroogeMD22]

But if I exclude a case for a certain variable, shouldn't I exclude this case for all other variables too?

[@Arch0s]

I have been looking for an answer to exactly this for some time now. My logical thoughts on it are that it depends on why you are removing an outlier, as you have probably read you shouldn't just exclude because they exceed a standard deviation too many. You need to look at your data and consider if the outlier is a product of an error in data recording for example I suspect you could just remove that one data point.. e.g. height recorded as 182m which is impossible but reasonable if in CM - recording error. Perhaps with social sciences there were confounding / external variables that impacted upon that particular cases' data. If that is the case I suspect the correct method would be to remove the case entirely as you could argue that the external factors may/would have affected all of the data. Would very much welcome any other thoughts on this.

[@MrKKrid]

how i define PC mom column? It use mean or something ?

[@KittyFIxX]

i love the deleting technique :)

[@SiobhanPhD]

It's not my go to move - but sometimes it's the easiest way. A quick fix when I just need to move on!

[@vkunst85]

Thank you for your video, do you perhaps know who I can cite for this method?

[@novitaasastr]

hello i want to ask, if i want to do the run test, i have to use the new data or old data that still have the outlier? *sorry for my bad english

[@ivantau9753]

I believe, once finding the outlier you need to just remove it from the existing data. Imagine someone conducting an experiment needs to throw all his data just bcs of the outliers, that would be ridiculous....If I understood your question correctly...

[@whiteshadow59]

thank you very nice explanation, been frustrated all day, thanks

[@chunnjh94]

Very useful video! Thanks for uploading!

[@chiangtongseng777]

what is PC_mom?

[@SiobhanPhD]

Truly, I don't recall exactly but it was something like Parental Compassion, which we calculated separately for Mom and Dad so we had a PC_Mom and a PC_Dad for that analysis and this video just shows the PC_Mom.

[@jamietan3288]

+Siobhan O'Toole PC_mon is mean or sum? is it a items or variable?

[@AbdulRazak-cz3rq]

yes, I want to know that too +Siobhan O'Toole what is PC_mom? the mean of the pc_mom1,pc_mom2.....etc?

[@weihuang6207]

Very helpful. Thank you so much!

[@SiobhanPhD]

Quite true.

[@SiobhanPhD]

But I'm a numbers girl myself!

[@acru518]

Thanks a lot

[@ShahbazAhmedENG]

Hoaglins method use 1987

[@azapura]

nicely explained but you assume too much of the average semi-educated student...

[@cahpintermail]

Hi, I wonder if you can give me some references on literature in doing this deletion on outliers.. Thank you in advance.