Why Dividing By N Underestimates the Variance

[@statquest]

Corrections: 3:23 I should have said "To understand why dividing by n underestimates the variation around the population mean". 3:40 The estimated mean was switched with the population mean. Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

[@Viralvlogvideos]

BAM BUM hahah

[@mayurihazarika6550]

Please Give Video on degrees of freedom please🙇

[@m3c4nyku43]

At around 8:35, you should've used asterisk '*' character instead of 'x' character for multiplication. I was a bit confused and thought you wrote 2*(x-v)*x-1 instead of 2*(x-v)*(-1). Great video by the way!

[@statquest]

@@m3c4nyku43 noted

[@paulpaschert6215]

is there some sort of award we can give this guy? please?!

[@statquest]

:)

[@jacobmoore8734]

I think we're encouraged to purchase a double dam t-shirt or sweatshirt, which is more of a financial incentive than an award but who doesn't like getting paid to be awesome? I'll probably pick one up this weekend

[@statquest]

@@jacobmoore8734 Thanks! :)

[@paulpaschert6215]

@@jacobmoore8734 award + t-shirt = double bam! just ordered my own shirt. gonna wear it to my statistics test in 2 weeks

[@karankartik1327]

Really, your way is too unique. one of the best

[@arun5351]

Amazing Josh!! I can't imagine how much hard work goes into simplifying the complex statistics concepts and coming up with these amazing videos. And on top of that your ingenious ideas of adding humor and musical creativity, taking the content to another level. If there was an Oscar for tutoring you'd be the undisputed winner. BAMMM !!- simply the best educator on KZbin....

[@statquest]

BAM! :)

[@ksrajavel]

Came to this video for "Why Dividing By N Underestimates the Variance" but got to know why absolute values are not used in Variance calculation. Literally cried, Prof. Josh.. Kudos to you. You are supporting me to understand the topics in statistics. I will support you regularly after I get a job soon. And I'm sure your teachings are required for many of the upcoming students in the coming decades. In India we have a concept called "Guru Kulam", and I see you as my guru (Not the term commonly known in the western world, this is more about respect)

[@statquest]

Thank you so much!!! It means a lot to me.

[@namanjain8939]

I searched for this on a number of online resources, some mentioned "n" while others "n-1", leaving me confused. This is the best possible explanation to the problem you made it really easy for us to understand. Thanks a lot !!! Bammmm subscribed and shared with friends.

[@statquest]

Awesome!!! Thank you very much for subscribing and sharing my videos with your friends. :)

[@ramkotha4726]

Josh, This is a total hypnotism you did with BAMs, echos, and other sounds. You mastered the art of making us stick there. I been searching for statistics and machine learning videos where they have kind of a roadmap, and simple explanations for complex topics, and this is it. You saved my life for sure, my donation is on its way, I know anything is small for what goes into making these. Hats off to you, you are a LEGEND. We owe you...

[@statquest]

Wow, thank you!

[@nursahidassafaat6283]

I've been 2 years asking how to plot variance, why sample variance (also sd) divided by n-1. And this is best explanation i ever had

[@statquest]

Awesome! :)

[@arbanafal]

I have nothing but admiration; this is the clearest explanation that I've seen so far that does not shy away from the underlying math, yet still keeping it understandable for those with minimal math background. I feel like a bit of a fool when I see the contrast between my own attempts to explain this correction factor and your explanation.

[@statquest]

I'm glad you like the video so much. Thanks! :)

[@punktdotcom]

I rather get a clear and understanding explanation with "BAMS" like i'm five, than a 50 pages long explanation with words like "trivial" and abbreviations (q.e.d) and just feel depressed and left clueless. And an other very important thing: Only if you *really* understood the topic, you can explain it with easy words. Very well done, Josh! Thank you very much!

[@statquest]

Thank you very much!!!! :)

[@dimiw5435]

the best accessible explanation I can find in the whole internet for this mystery. then just as I was about to say "aha! you missed out something!" towards the end of the video, you seemed to have read my mind and "p.s. if you are wondering why n-1 and not 0.5 or 2 .... " you are so so spot-on!

[@statquest]

Thank you very much! :)

[@naysannaderi5135]

@@statquest I agree - best explanation i have found and i'm sharing this video with all my students. THANK YOU! So.... any chance that next video is coming out soon? (or has come out already?)

[@statquest]

@@naysannaderi5135 I hope the next video will come out soon. Possibly in the next 4 months or so. I hope!

[@davidh1876]

Big thanks from Taiwan. I have been asking why not dividing by n since high school...but all I get from my teacher was only "a rule of thumb". Now I know the reason behind and thanks to statquest. BAM!!

[@statquest]

Hooray! :)

[@jsc3417]

Thank you, 10 years of confusion made clear by this 15 mins of video.

[@statquest]

Hooray! I'm glad the video was helpful. :)

[@ireneylhsiao]

Bam!!! I've watched lots of your videos after I discovered the one explaining the standard error. You make me understand stats concepts more clearly. Please continue making these awesome videos (machine learning too)! 5 dollars donated!

[@statquest]

Thank you very, very much. I really appreciate it. :)

[@wowZhenek]

Yet another video from this channel that leaves me speechless. I've never really understood this concept until I've watched your video. Thank you very much, again.

[@statquest]

Wow, thank you!

[@achannel9598]

came from calculating the mean, variance and SD video. Did not expect a proof for why variance = x-bar. This is a really good in depth video i've ever watched for statistics. Thank you very much.

[@statquest]

bam!

[@Ana-wx8jm]

I click the like button before I watch it because I'm always sure I'll love it! Thanks so much for making this series. You'll never know how helpful it has been in my life

[@statquest]

Hooray!!! Thank you very much! :)

[@killua9369]

I have always hated statistics but I just today found this channel and this guy explains everything elegantly! ❤😊

[@statquest]

Wow, thank you!

[@GbUnLimiteD]

Yes! I had already feared that the n-? question won't be explained. Glad to hear that you will explain this unsolved mystery in the next video!

[@statquest]

Unfortunately, it will be a while before I get to it. I've got covariance and correlation coming up next, then a few machine learning videos, but then I'll loop back to expected values. It's a topic that I've wanted to work on for quite some time.

[@2oqp577]

My uneducated guess about n-x is that the bigger the magnitude diff. between the population and your sample size, the larger x would be. Because as this magnitude get smaller and smaller, the need for x to have any significant value, disappears. My biggest question is why would x lead to this unitary value when your sample size is little. But we'll see what Josh explains about that.

[@nickp7526]

Intuitively: the number you're dividing stands for the degrees of freedom you have. In other words: how many data points are allowed to vary freely. The reason that this is 1 less here is, as the video hinted at, because of the sample mean. If someone shared with you n-1 data points of their sample distribution of n points, and you know what the sample mean is, then you can easily calculate what the last data point is. I.e. that last data point doesn't have any freedom to vary, just because it was crucial in defining the sample mean. This doesn't matter if you know what the population mean is, precisely because the sample distribution didn't decide its value. Therefore all n values in a sample distribution with known population mean can be used to make an unbiased estimator, while only n-1 degrees of freedom can be used to have an unbiased estimator when all you known is the sample mean. Mathematically: en.m.wikipedia.org/wiki/Bias_of_an_estimator The first example (in the examples tab) shows why it should be n-1, and not n, or n-whatever.

[@libertarianPinoy]

Kids today are so lucky they can review their stats online like this with great teachers.

[@statquest]

:)

[@chathurijayaweera1590]

Thank you for this explanation. When I was learning stat in university, I did not understand well, why we divide by (n-1) instead of n to estimate sample variance. You explained it so clearly in a way that I will never forget what I learnt. Thank you Josh !!!

[@statquest]

Hooray! :)

[@keej7146]

Thank you for this!! The first time I saw the formula for the sample variance I wondered why the n-1 was there, this is a great explanation.

[@statquest]

Thanks!

[@Calypso-rt5tf]

hello, keej i hate u mate

[@taotaotan5671]

I just read wiki and found that even divided by n-1, we still underestimate the standard deviation (although we don't underestimate the variance anymore). I feel that's somewhat mind-blowing, since calculating sample std is such an ordinary job for statisticians, and it is surprisingly BIASED (and I am sure the standard error formula is also biased)...

[@statquest]

interesting

[@taotaotan5671]

@@statquest Yeah. This is the wiki page. en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation

[@mugssyy]

Michael Scott: Why don't you explain this to me like I'm five? Josh Starmer: Bammm!! and understood ... thank you : ) !

[@statquest]

BAM! :)

[@Deepak-uv8du]

@@statquest Can you provide the slides for all the statistics videos you used to explain the concepts

[@statquest]

@@Deepak-uv8du I have PDF study guides for some of my videos here: statquest.org/studyguides/

[@cristianleoni6852]

Amazing explanation of why we use the square of the errors instead of the absolute value! I always asked myself that and all the teachers said it was just to give a bigger weight to the errors! We need the statquest on expected value!

[@statquest]

Thanks! I'm working on the expected value, but it still might be a few months before it's ready.

[@theblinkingbrownie4654]

I think for even n there wouldn't even be a minimum point, rather a flat line between the 2 middle samples

[@scuti7073]

Man, I always thought that statistics doesn’t make any sense at all and that people should just blindly chug into weird formulas without questioning, but this was absolutely mind opening. Not even khan academy could explain the proof!

[@statquest]

Thanks!

[@lelamakharadze727]

"Future is nooow, BAM " - #LOL #respect #welldone #thanks

[@statquest]

Thank you! :)

[@Michael-zn4oq]

Thank you so much for the clear and simple explanation. This is an example for when showing the proof is better than only trying to give an intuition.

[@statquest]

Thanks

[@anujlahoty8022]

Awesome and the best video with most simplified explaination.

[@statquest]

Thank you! :)

[@magtazeum4071]

8:22 `the way he said "Whaat" is so cute.. I'm in love

[@statquest]

:)

[@haoqichen7610]

The last point about absolute value explains a lot! I was always wondering why squaring data is so much more common than taking absolute values!

[@statquest]

bam! :)

[@anishchhabra5313]

This is epic, never got a better or clearer explanation for this particular problem. Hats off!🙌

[@statquest]

Thanks a ton!

[@Drugio24]

this is literally what I was trying to get a clear understanding on in the last few days? what are the chances? no seriously what are the chances?

[@statquest]

That's awesome! :)

[@tumul1474]

Statquest, JBstatistics and Khan Academy.....You guys are just amazing !!.....Thank you for all you have done for us

[@statquest]

Thank you! :)

[@izebit]

Thank you, I haven't known about these channels

[@yildizkoca8878]

This video is such a gem! Thanks for explaining the root of this concept which is not easy to find even in statistics books.

[@statquest]

Glad it was helpful!

[@PraveenKumar-yv5zn]

This is the best explanation that I've come across for this. And I really liked that you gave a proof for general set of observations. Thanks a lot.

[@statquest]

Awesome, thank you!

[@nizarch22]

I don't even remember what I was confused about in particular, but I remember feeling very happy to see this video. Will revisit this in the following days. Psst, you're a gem ;)

[@statquest]

Thank you very much! :)

[@christopherchen4920]

The most impressive explanation I've ever seen.

[@statquest]

Thanks!

[@tippyandfriend]

This is excellent, I am looking forward to the next one.

[@Ujjwalchhabra1]

You left in a cliff hanger of expected values :(( Love your videos tho, thanks for these!

[@statquest]

I'm working on it, but everything I do takes longer than I would like. :)

[@nidhiarora4739]

I have been SO stressed out about a project I'm working on, and 3:15 made me laugh so hard!!! I didn't even realize how stressed out I was until I caught myself laughing for the first time in weeks. Thank you Josh!!! **sob**

[@statquest]

Hooray!!! Good luck with your project. I hope it goes well. :)

[@alexandermedina4950]

I can only have love for these videos, thank you Josh and all the team if you have any.

[@statquest]

Thank you! It's just me doing all this.

[@morenomartinovic4385]

I'm eagerly awaiting the expected values quest! Thank you so much for making these videos, I love watching them before sleep.

[@statquest]

Awesome! It's on the to-do list, but it might not be done for awhile. :(

[@morenomartinovic4385]

@@statquest That's cool, take your time to keep making awesome videos. I still have loads of your videos on my to-watch list!

[@coldbrewed8308]

Oh no... I'm falling deeper and deeper into this rabbit hole

[@statquest]

:)

[@iAmTheSquidThing]

This intuitively makes more sense to me now. If I take a sample, the sample mean may end up being larger or smaller than the population mean. But the sample variance can never be larger than the population variance, it might be equal to it, but most probably it will be smaller.

[@statquest]

That's exactly right. :)

[@wobwobvoid420]

I think you have to be a little bit careful with what you mean by "sample variance" and "population variance". As long as you're comparing an estimated population variance using the sample data and actual population mean vs an estimated population variance using the sample data and sample mean. But, comparing the estimated population variance using the sample data and sample mean vs the actual population variance (all data and actual population mean) doesn't have the guarantee that sample variance will be lower than population variance.

[@emmaning992]

I admire this explanation... Amazing. I really look forward to the expected values video!

[@statquest]

Thank you. I started working on the expected value video, but it will still be awhile before I finish since I have many other projects to work on.

[@ujjwal2912]

Although you make everything look so simple, your teaching pedagogy requires a lot of hardwork( to make the slides particulalry). I hope that every teacher puts in the same kind of hardwork and assume there students to be in 5th grade that way every class will be a pleasureable experience of life.

[@statquest]

Wow, thank you!

[@waddragon]

it is because some teachers don't know how to teach. They learn from textbook's concept. Memorize them, then give those back to students. I am not being rude but it is the reality. In order to be able to explain well to new learners, teachers must be able to understand the concepts well. Teaching is a hard skill to master. Nowadays, lot of taught concepts are assumed true or left blank during teaching . That's why if those students become teachers, they won't be able to explain.

[@ARM26878]

BAM! I have not seen this concept explained better anywhere else ever. Have you gotten around to making the follow-up video on 'expected values' ? Can't thank you enough for your channel

[@statquest]

I've got the video on expected values kzbin.info/www/bejne/gX3WkGqYbLh-n5Y and kzbin.info/www/bejne/hYSzo2l9a7CUY7c , but there are still a few steps to go after that... :(

[@Patrick_Bentolila]

Clarity brings understanding

[@statquest]

Bam! :)

[@lyrachang950]

im currently learning data analytics and trying to figure out ab testing and bam! here i am! thank you so much for making statistics fun and easy to understand! double bam!

[@statquest]

Happy to help!

[@keysky_1622]

wow that n-1 has something to do with E(X)? Im waiting for it!

[@edward8064]

Mind = Blown. Thankyou from Indonesia.

[@statquest]

Thanks!

[@Marius-vw9hp]

I wanted to know why we deduct exactly 1, but I guess that only takes 20 aditional minutes to explain. Hooraay! Thanks for the videos :)

[@statquest]

It's true. We have to dive into expected values and that is a whole new topic.

[@rajarshibasak347]

Aah! Finally end. What a excellent work by you!! Statquest rocks ❤.. Thank you sir. You helped a lot in my carrier ❤.

[@statquest]

Thanks!

[@ryanmckenna2047]

This channel is just incredible, well done!

[@statquest]

Thank you very much! :)

[@NuclearSpinach]

"The future is now" I'm dying

[@statquest]

BAM! :)

[@ThalesBrunoM]

8:21 -> I will watch a thousand times and I will laugh out loud a thousand times 😂

[@statquest]

Hooray! :)

[@Igor-vb1hv]

Thanks for explanation! I understand that differences between the SAMPLE data and the sample mean are smaller than the differences between the SAMPLE data and the population mean. BUT! We are not interested in the difference between the SAMPLE data and the population mean, rather we are looking for the difference between the TRUE POPULATION data and the population mean (the population variance). And it's not clear why this value would be larger. I mean sample data is centered around sample mean the same way population data is centered around population mean. Comparing sample data with population mean feels to be misleading.

[@statquest]

The best estimate we can do is the estimate of the variance around the sample mean, which is probably an underestimate, but not always. So this is the best we can do.

[@brucewayne6744]

Great explanation!! I'm loving every second of your videos!!! Cheers!!

[@statquest]

Thank you! :)

[@yufeizhan726]

I finally know why n-1 is used. Thank you so much!

[@statquest]

Bam!

[@thepahadiboi]

What a explanation. I don't have money, else I'd have contributed. The least I could do is share, which I already did. BAM !

[@statquest]

BAM! :)

[@HamidNourashraf]

I love the way you explain these topics, great work!

[@statquest]

Thanks!

[@mansoorbaig9232]

This is awesome explanation. Waiting for quest on 'Expected Values'....BAM!

[@statquest]

Me too. Hopefully I can get to it soon.

[@MrBlissTube]

Great video! Where is the one about Expected Values? I cannot wait with such a cliffhanger! GoT finale can wait...

[@statquest]

Very funny! Yes, I have my work to do. I hope to get to expected values before too long.

[@MrBlissTube]

@@statquest Thanks a lot for responding! ... and sorry, as I noticed after reading more comments, that you had already answered this question many times. Quest on!

[@OdysseusKingofIthaca-o4n]

Thank you St Josh for this illuminating explanation :)

[@statquest]

My pleasure!

[@exoticcoder5365]

15:46 it’s the god moment 👏🏻👏🏻👏🏻👏🏻

[@statquest]

bam!

[@ineshtickoo2530]

other yt channels: gets banned statquest: gets BAMMMED!

[@statquest]

BAM! :)

[@ginopeduto4264]

THX!!! Looking forward for the STATQUEST on expected Values ;))))

[@statquest]

Me too!

[@ymlow91]

I still dont really understand why showing V = X bar at minimal variance leads on to the conclusion that we always underestimate the population mean by dividing by n

[@statquest]

What part of the video is confusing? Can you specify a minute and second time point?

[@sephirothjc]

This the best explanation ever

[@statquest]

Thank you!

[@marinasha2949]

Good job Josh!! Waiting for StatQuest on Expected Values! I am the one wondering why not dividing by 'n-0.5' or 'n-2'

[@statquest]

Thanks!

[@stevemungai3542]

The minute you mentioned derivatives, my mind shut down!

[@statquest]

:( They sound fancy, but they aren't that bad. If you want to learn more about them, see: kzbin.info/www/bejne/rZ2Unqyup9mEfrM

[@ROTOBAfilms]

You are a very great teacher, i like your coaching style, keep going on!

[@statquest]

Thank you! 😃

[@shashankupadhyay821]

I usually hit like after the first BAMMM. This is some super great stuff Josh.

[@statquest]

Thank you very much! :)

[@mukhtarbimurat5106]

Greatest explanation so far!

[@statquest]

Thank you! :)

[@sunaxes]

Waited the whole video to know why it was n-1 and not n-2 etc... "that mistery will be resolved in the next episode" Felt like watching an overstretched TV series in some way haha. I understand it's capital to show first that sample variance underestimate true variance, but could mention earlier that you ll not focus on "why" it is n-1. :p Thank you though wonderful content!

[@statquest]

I tried to be careful with the title of this video with "Why dividing by N underestimates the variance" instead of "Why n-1 gives us an unbiased estimate". That being said, I really wanted to explain exactly why n-1 works, but the proof is relatively advanced.

[@truewarrior911]

I literally watch your videos for fun.

[@statquest]

bam!

[@vkvkvkvk]

baaammm! subscribed.

[@statquest]

Awesome! :)

[@nursahidassafaat6283]

BAAM! me too

[@shouryanand456]

I wish you were my stats teacher!! Amazing job!!!

[@statquest]

Thank you! :)

[@shouryanand456]

@@statquest really waiting for the expected value video to get explanation of n-1. When can we expect it?

[@statquest]

@@shouryanand456 Unfortunately, it might be a while. I've got a full plate until after the summer.

[@thegamingannex5752]

Your work is impeccable. BAM!

[@statquest]

Thank you!

[@fernandomendoza9372]

Este pibe tiene futuro.

[@statquest]

Muchas gracias! :)

[@Lsazeh]

Thanks so much for the explanation, super clear as always

[@statquest]

Glad it was helpful!

[@wexwexexort]

Waiting for StatQuest on Expected Values.

[@waddragon]

thanks for great explanation of the minimum value of the curve is the sample mean. However, the explanation of using absolute is not complete. A better way to explain it is that if we use absolute, we won't be able to tell how well the data is spread from the mean than using the square. The reason is coming from the case A when data has even space from the mean and the case B when some data has very large space from the mean but the rest has small space from the mean. If both case A and B give same result variance value from absolute formula , then we won't be able to tell accurately how well data is spread out . However , by using square, case B will give bigger variance than case A. That's why square is preferred. It is not full proof but i think it provides better intuition than the straight lines instead of parabolic slope 's explanation.

[@statquest]

Thanks!

[@radosawszostak6104]

Great video! We clearly see that estimated variation is smaller than desired so we have to make it bigger. We can make it by dividing by n-1, but also by n-2 or n-1.5 or n-100. Why n-1?

[@statquest]

One day I'll make that video, for now, see: online.stat.psu.edu/stat415/lesson/1/1.3

[@shubhamtalks9718]

Man, you are great. From where did you learn these concepts? Keep making videos and enlighten us. Thank you.

[@statquest]

Thanks! :)

[@shubhamtalks9718]

@@statquest When I try to learn these concepts they seem complicated to me. From where did you learn these concepts?

[@statquest]

@@shubhamtalks9718 The concepts seem complicated because people that do not really understand them try to teach them. How did I learn them? Years of really hard work. I read everything I can about a subject, then I re-read it. Then I re-read it again. Then I make a program based on my ideas and see what happens. Then I re-read everything over again. And sooner or later I figure it out. But it takes a lot of time and a lot of work. Sometimes I worry I will not succeed, and sometimes I fail, but I keep trying anyway.

[@shubhamtalks9718]

@@statquest Thanks😁

[@ipmankus]

Very nice explanation, god bless you josh!

[@statquest]

Thank you! :)

[@fernunez2131]

I love the humor included in this videos, hahahah!

[@statquest]

Thanks!

[@eddy147Tennis]

Question: why is it that the differences between the data and the sample mean is less than the differences between the data and the population mean? If I take the most extreme data points for the sample mean won’t I have a bigger variance for the sample data? P.S. I agree you need to get some award, videos are fun, educational and clear.

[@statquest]

One of the properties of the sample mean (which was illustrated in the examples and in the calculus) is that it always in a location that minimizes the distances between it and the sampled data.

[@koreanbroadcastarchive306]

Excellent. Thank you for a great explanation.

[@statquest]

Glad you enjoyed it!

[@ankitvyas9013]

This is gold.

[@statquest]

Thank you!

[@haugstve]

Nice! PS. Small typo at 8:42, you say -1 but write x-1.

[@statquest]

That 'x' is a "times" symbol. So it's "times -1", not "x - 1"

[@ginopeduto4264]

@@statquest thx I was confused too - by the way - your videos among the very best one can find!!! Thank you so much!!!

[@GurunathHari]

Brilliantly explained....

[@statquest]

Thank you very much! :)

[@honghur8977]

One question is left out: Why is underestimating the variance with sample mean a problem?

[@statquest]

If we underestimate it, we will have more confidence than we should in our estimates or predictions of future events. Typically, you do not want to error on the side of over confidence.

[@hafidhrendyanto2690]

Amazing video! I think that you should teach another subject. Maybe MathQuest? That would be amazing!

[@statquest]

Maybe one day!

[@kanundash]

waiting for expected value episode :D

[@statquest]

Me too!

[@Lets_MakeItSimple]

Hey josh, yet another cool explanation.

[@statquest]

Thank you! :)

[@Shakey719]

Cool as Ice has been by commentaried by Rifftrax. I would watch that...but I would watch StatQuest, too.

[@statquest]

That's awesome!

[@claudelorrain-bouchard6941]

dang... why did I end up watching cool as ice (starring vanilla ice) instead of statquest!

[@statquest]

:)

[@chiragsomani101]

ASTOUNDING EFFECTS & EXPLAINATIONS! SUBSCRIBED TRIPLE BAM!!!

[@statquest]

bam!

[@alishashingade4456]

You are so awesome ! Thank you Josh :)

[@statquest]

Thank you! :)