Expected Values for Continuous Variables!!!

[@statquest]

Support StatQuest by buying my books The StatQuest Illustrated Guide to Machine Learning, The StatQuest Illustrated Guide to Neural Networks and AI, or a Study Guide or Merch!!! statquest.org/statquest-store/

[@theburtmacklin9615]

Amazing videos; if you only knew the way in which your channel had changed things for me… what started as a “hey this video looks cool” has largely been responsible for my wife enrolling into a data science postgrad program in the University of Waterloo.

[@statquest]

That's amazing! Wish your wife good luck for me and I hope it goes well. It's very exciting! BAM! :)

[@Thedavidk]

@@statquest I did but the integral stuff was a bridge to far for me 🙂

[@nichomajin636]

Why do I have to attend college classes when you explain these concepts so thoroughly and in a third of the time? Great video as always man you are a Godsend

[@statquest]

Wow, thanks!

[@raulsena3917]

Amazing videos, just wanted to thank you for all you have taught me. Now I am doing a Master's Degree in Artificial Intelligence and planned to do a Ph.D. and all thanks to your videos.

[@statquest]

Wow! That is awesome! Good luck!

[@bogdankhamelyuk3857]

same bro. Also doing MS in AI and reminiscing some ML basics XD

[@norukamo]

Just discovered this channel an hour ago and now I'm binging your videos. They're soo good and easy to understand. BAM!

[@statquest]

Thank you very much! :)

[@ahmadatta66]

I'm in the library and I did not realize my laptop speaker was on, and the video started, as always, with a song. :D I'm not even embarrassed. Everybody should hear your songs, they are amazing haha

[@statquest]

BAM! :)

[@otavioaugustodeassuncao9858]

Currently i'm studying Statistics at Federal University of São Carlos (UFScar) and just wanted to thank you for all the helpful and fun content that you've been posting... Not only had helped me to understand but also has made me like Statistics even more

[@statquest]

Thank you very much! :)

[@astar2128]

Most of my knowledge on stats and data analysis is self taught and this channel is a godsent (infinite bam!). Now i have to learn some calculus😅. Math has always intimidated me but I want to understand the concepts.

[@statquest]

bam!

[@zalavadiaridham]

Amazing video josh!!! I'm waiting for the video on why we divide by n-1 when we compute the sample variance. Thank you for the very informative content that you put out.

[@statquest]

Thank you!

[@hrig]

yes! n-1please!!!!!!!!

[@jaspermuller9851]

Is there any update on the n-1?

[@dicksang2]

I am following through and want your version of n-1 as denominator to estimate the variance by sample variance. Thanks!!

[@durgeshkshirsagar116]

You sound like kids' rhymes but when you teach, teach like a pro.

[@statquest]

Thanks! :)

[@durgeshkshirsagar116]

@@statquest welcome

[@josherickson5446]

Triple bam bro! Now just need to refresh some calculus :)

[@statquest]

Yep!

[@MrDanituga]

One of the best explanations I have ever had! Congratulations on the video and on your content in general. Watch every video :D

[@statquest]

Thanks! 😃

[@fedos]

I think I've had too much to drink, because I read the notification as "Star Quest with Josh Starmer", and I was confused about your sudden shift to astronomy. I'm going to have to watch this tomorrow.

[@Erosis]

Star Quest would make a great April Fools joke. Have any astronomy friends, Josh?

[@statquest]

Once StatQuest makes me a billionaire, I'll make spaceship called Star Quest!!! You guys can come along for the ride.

[@jerrycheung8158]

Great video! May I ask where is the next video to go to for solving the n - 1 puzzle?

[@statquest]

Unfortunately I haven't made it yet. However, this is my favorite webpage on the topic: online.stat.psu.edu/stat415/lesson/1/1.3

[@trad_sikh]

@@statquest I have a question which is bothering me. We know sampling distribution’s standard deviation (standard error) times square root of sample size is the population’s standard deviation. Why are we even getting into this mess of dividing by n-1? Is it because the first case is for a sampling distribution (distribution of means of samples) while the later is a simple random sample ? Am I mixing Central limit theorem where it doesn’t belong?

[@statquest]

​@@trad_sikh To see why dividing by 'n' underestimates the population variance, see: kzbin.info/www/bejne/qXm1c5psd81-oKs

[@awaisjabbar8304]

Liked the video, before it even starts, awesome channel, why I have not heard of you before.

[@statquest]

Thank you!

[@sparshasherke250]

this explanation is magical :cries:

[@statquest]

:)

[@newsupdates3622]

Double bam! Here to watch another great statquest video for a refresher

[@statquest]

Thanks!

[@homunculide8567]

Thanks for that and another videos with all that useful explanations... I'll keep binge watching. Regarding your explanation at 10:06... The second term and all following appear wrong to me, because the intervals are 10s each and not 20s, 30s, 40s... this the area of the rectangle should be likelihood x 10(s) for all terms (discrete rectangle) You are the expert but I can't wrap my head around that.

[@statquest]

I'm not sure I fully understand your equation. The area of each rectangle (the probability) is 10 x likelihood (obviously the values are approximated and rounded in the figure). The time, 's', should only be used to determine the likelihood at a specific point.

[@selva279]

Eagerly waiting for your book...

[@statquest]

Working on it! :)

[@stef1790]

Thank you! Really helped with refreshing my statistics!

[@statquest]

Glad it helped!

[@alexandersmith6140]

Hi Josh. What video(s) after this should I watch to get closer to understanding why we divide by (n - 1) when finding the sample variance and sample covariance? An intuitive explanation for this seems nowhere to be found across the entirety of the internet, and the StatQuest channel has thus far been a divine gift of comprehension.

[@statquest]

Have you seen this one: kzbin.info/www/bejne/qXm1c5psd81-oKs Other than that, the best I can do is refer you here: online.stat.psu.edu/stat415/lesson/1/1.3 One day I turn that page into StatQuest, but not for a while.

[@atg6174]

@@statquest why not now? i think its time to do so ...😢

[@quanghoang3801]

@@statquest i have waited for 4 years 😢

[@Letsbetog]

You are an awesome singer As well as A great Teacher.🙃🙃🙃

[@statquest]

Thanks!

[@kakashisensei8146]

Josh, I'm really thankful to all your vdos, it's enlightening, lol. Please can you make some vdos on distribution functions (e.g., normal distribution)?

[@statquest]

I've got a basic video on the normal distribution here: kzbin.info/www/bejne/qKupiWiEjNSdppI and a video on maximum likelihood estimation with the normal distribution here: kzbin.info/www/bejne/ep-Zk2yceK6Ipq8 All of my videos can be found here: statquest.org/video-index/

[@kakashisensei8146]

@@statquest you're the best.. Thanks a lot!

[@sharan9993]

When can i know why we divide sd by n-1? I hav seen ur videos on sd and know we need to subtract something but why 1? U said we need to know about expected values first. So when can i learn? Love ur videos. Thanks a lot

[@statquest]

Believe it or not, we still have a long way to go. Sorry this process is so painful!

[@statquest]

That said, if you want to skip to the head of the class, see: online.stat.psu.edu/stat415/lesson/1/1.3

[@sharan9993]

@@statquest ohhh i will wait bcoz i am curious yet lazy😂

[@sharan9993]

@@statquest i checked it out and it went over my head. Could u tell me the path that leads there? I will search and read them. Thanks a lot

[@statquest]

@@sharan9993 If I could, you wouldn't be waiting for my videos.

[@eduquest3169]

Thanks a lot dear, may god bless you Josh.

[@statquest]

Thanks!

[@FortranCastle]

All this integration and derivation and by the end, the expected value (quite intuitively) was the reciprocal of the rate. I mean it makes a lot of sense without needing to prove it. If you expect to see 5 people per minute, you can expect to wait 1/5 minutes between encounters on average.

[@statquest]

Yep! The idea wasn't to blow your mind, but to show you how a process works using an example that you can easily verify with other means.

[@Altamira_]

Thanks for all of your videos they are great. I'm waching the full playlist "statistics fundamentals" and have to admit that since I understood all of the previous videos thanks to your great explanations I was lost on this one when you used integrals :/ I think it's the video that requires the more calculcus knowledge.

[@statquest]

As long as you understand the discrete case, you should be good to go.

[@Altamira_]

@@statquest Thanks :)

[@egeilkay8882]

when the statSquatch said "TRIPLE BAM" it just hit different

[@statquest]

:)

[@adampotter760]

What the next video to watch to find out the elusive reason for dividing by n-1!? Please reply!

[@statquest]

Unfortunately I don't have the video yet. In the mean time, check out: online.stat.psu.edu/stat415/lesson/1/1.3

[@НиколайТолстов-м3и]

1) Why cant we just take average of all waiting times to get how much on average we would wait? 2) How often in real life we can use some sort of formula like exponential to describe a distribution? I think in most of the cases we can use NN to fit the distribution

[@statquest]

Expected values are helpful for doing Statistics and getting a sense of how likely future events will be. This is why we go through the trouble to do all this math instead of just taking an average or fitting an NN to the data.

[@samieweee7468]

I am a bit confused is the value in y axis the likelihood of meeting a person after xth seconds or the number of people we meet after x th seconds cause during initial explanation the dots at each x were number of people we met after xth time.

[@statquest]

What time point, minutes and seconds, are you asking about?

[@samieweee7468]

​@@statquest 2:37

[@statquest]

@@samieweee7468 At 2:37, each dot represents an individual person. In other words, StatSquatch is creating a histogram. However, histograms have problems - gaps in the data and the data are not continuous (they have to be put in bins). Thus, we use an exponential distribution to approximate the histogram. The exponential distribution doesn't have gaps and is continuous. And, from there on out, we use the distribution, which gives us likelihoods on the y-axis, rather than the histogram, which gives us the number of people on the y-axis.

[@AG-cx1ug]

Did we finally get the final equation as at 7:40? Both are just area under the curve? Width wasn't being used before 11:00 - we were just using the specific outcome x, how did both width and specific outcome come in the final equation?

[@statquest]

The equation at 7:40 is just the equation for the area under a curve defined by an exponential distribution, not the equation for the expected value of an exponential distribution. The equation for the expected value of an exponential distribution doesn't show up until 12:39. The big difference is that we multiply the formula for the exponential distribution by 'x', a specific outcome. Width isn't part of this equation because we take the limit earlier as the number of rectangles goes to infinity and their widths go to 0.

[@DjamelEddineStudentAchour]

amazing, thanks for the video.

[@statquest]

Thanks!

[@AG-cx1ug]

At 11:13 why is there still an x value at x * L(X=x) * width? Wasn't the x the width?

[@statquest]

'x' is the value on the x-axis. It refers to a specific outcome or event happening at a specific time.

[@tauqeeralitapya6447]

at 9:39, why are we multiplying 20 with 0.2 and not 10 as the interval is still 10 between 10 and 20?

[@statquest]

20 represents the specific, discretized, outcome. In other words, we have discrete time points, 10 seconds, 20 seconds, 30 seconds, 40 seocnds, etc. that we have to wait before we meet someone in StatLand. Since we have a histogram, everyone we meet between 0 and 10 seconds is categorized as "someone we met in 10 seconds". Likewise, everyone we meet between 10 and 20 seconds is categorized as "someone we met in 20 seconds". etc.

[@priyankjain9970]

@@statquest then it should be like = (10*0.4) + (20*(0.4+0.2)) + (30*(0.4+0.2+0.1)) and so on.. and that's because for x=10 , probability will be area under curve till 10 which is 0.4, and for x=20 probability will be area under curve till 20 which is 0.63(roughly equal to 0.6) ... Please correct me if I am wrong

[@drelijahmikail3916]

shouldn't it be a Riemann sum where the interval is the same? Why is the multiplier: 10, 20, 30, etc, instead of constant interval of 5 or 10?

[@statquest]

Because the x-axis value (10, 20, 30) represents time and we want to know the probability of observing something during that block of time.

[@mohammedk.h.f3016]

Thank you very much

[@statquest]

bam! :)

[@SunSan1989]

I watched all the videos carefully, but why can't I find it? Why choose n-1 instead of N minus 2 or n-0.5?

[@statquest]

Unfortunately I haven't made that one yet. In the mean time, check out: online.stat.psu.edu/stat415/lesson/1/1.3

[@AG-cx1ug]

For each rectangle shouldn't we be multiplying the likelihood value by 10 and not 20,30? Since the area is just between 10-20 or 20-30?

[@statquest]

At 9:00 we calculate the y-axis values (the likelihoods) for each rectangle by 10 because it is the distance between each tick-mark on the x-axis. This gives us the probabilities of each event. However, the events, or outcomes, themselves, are the x-axis values. So, starting at 9:10, we multiply the probabilities by the outcomes (the x-axis coordinates). When we add these products up, we get a weighted average of the outcomes (weighted by their probability of occurring). Is that what you are asking about?

[@গোলামমোস্তফা-শ৮থ]

Probability =( height*weight). Of rectangle How did you find that? Can you please explain how it indicates the probability? .. and also explain me that what is the value of "x" in continous random variables "mean"?

[@statquest]

The area under the curve between two points represents the probability of something happening between those two points (see: 4:48). This is simply how probability distributions are defined. We can solve for that probability exactly using calculus (see: 4:49), or we can approximate it using rectangles (height * weight).

[@mamadi-uk7kv]

Perfect man,Great

[@statquest]

Thanks a lot!

[@maxyen9892]

Could you comment on how we would find expected values for a normal probability distribution, given a mean and standard deviation?

[@statquest]

The expected value for a normal distribution is its mean.

[@mandaltech6249]

Waiting for your book. You area real teacher.

[@statquest]

Thanks! I hope it is out in May!

[@isurumahakumara]

I couldn't find any quests on Time Series, If there aren't any would love to see one in the future!!!

[@statquest]

Me too! BAM! :)

[@amirhosseinshafieian3951]

THANK YOU for your great videos I just have a question, I think( I am not sure) that this data in this example follows Poisson distribution, not exponential!!!!! , am I right???

[@amirhosseinshafieian3951]

I think I got it my self

[@statquest]

Poisson is discrete and models something very differently from what we are modeling here with an exponential distribution. For details, see: en.wikipedia.org/wiki/Poisson_distribution

[@VladLanz]

Looks like the math is wrong for the approximation of expected value of distribution: 1) for each bin we compute: f(x) = lambda * e ^ (-lambda * x), where f(x) is the value of probability density function (PDF) for the middle point 2) we compute probability of each bin: p(x) = f(x) * delta, where delta is the width of the bin, in our case = 10 3) This is a step with mistake: we calculate contribution for each bin and sum everything up: E(x) = sum ( p(x) * x ), where x is the middle/average point for each bin, but in your video you took upper bound instead of an average value for each bin. If you do calculation this way, you get 19.23 which is closer to true value

[@statquest]

I wouldn't say the math is wrong because the purpose is only to illustrate a concept, rather than how the math is actually done. In practice, we don't do a summation, we take the integral.

[@VladLanz]

@@statquest sure, in practice we take the integral. But for approximation it makes more sense to take average value for each bin rather than it's top value. [5, 15, 25, ... 95] instead of [10, 20, 30, ... 100]. In case that's you and not some hired assistant who's answering comments here: thank you for your work, you're amazing🤗 You're the main reason i managed to remember everything i learned in university more than 10 years ago, started to master ml and deep learning and began working as data analyst

[@statquest]

@@VladLanz That's me! Thanks! :) (and I still wouldn't say taking the edge is 'wrong' - different, and maybe it doesn't make as much sense for the sake of getting the best approximation, but not wrong).

[@6872elpado]

I didn't know you could call the integral of g'(x) the anti-derivative. Why didn't you just integrate g'(x) to get g(x)?

[@statquest]

"Antiderivative" and "Indefinite Integral" are synonyms. So we can say it either way. en.wikipedia.org/wiki/Antiderivative

[@oraciopozzo8694]

When you approximate the expected value it is confusing that you add areas whose base is not the constant 10 interval. The areas of the rectangles should be in your example variable height given by the formula times 10 and not 10, 20, … Otherwise the rectangles are meaningless.

[@statquest]

I'm sorry that was confusing to you. However, each rectangle has the same width (as you can see in the illustration). However, each rectangle represents a different specific outcome. So 10, and 20 are not different widths, but different outcomes. 10 is the outcome represented by the first rectangle (with width = 10) and 20 is the outcome represented by the second rectangle (also with width = 10).

[@priyankjain9970]

​@@statquest then it should be like = (10*0.4) + (20*(0.4+0.2)) + (30*(0.4+0.2+0.1)) and so on.. and that's because for x=10 , probability will be area under curve till 10 which is 0.4, and for x=20 probability will be area under curve till 20 which is 0.63(roughly equal to 0.6) ... Please correct me if I am wrong

[@statquest]

@@priyankjain9970 No, it's (10*0.4) + (10 * 0.2) + (10 * 0.1) + (10 * 0.09) + ... etc. To approximate the area under the curve, we add up the area of each rectangle. The area of each rectangle is the width (10) times the height (0.4 for the first, 0.2 for the second, 0.1 for the third, etc.).

[@priyankjain9970]

@@statquest Thanks for reply. Actually the height is 0.04 for first, 0.02 for second, 0.01 for third and so on ( as explained by you @8.29 in video). Therefore area of rectangle will be 0.4 for first, 0.2 for second, 0.1 for third and so on. My concern is following As you stated E(X) = Σ x * p(X=x) .. This means E(X) = 10*(probability till 10) + 20 * (probability till 20) + 30 * (probability till 30) and so on. Now probability till 10 means area under curve till 10 which is = 0.4 probability till 20 means area under curve till 20 which is = 0.6 (approx) probability till 30 means area under curve till 30 which is = 0.7 (approx) Therefore E(X) should be (as per my understanding) = 10*0.4 + 20*0.6 + 30*0.7 + .... Please help me to understand this

[@statquest]

@@priyankjain9970 Sorry about the typos with the area vs height. That said, the probability of observing an event between 10 and 20 seconds is not the cumulative probability of observing an event between 0 and 10 or between 10 and 20. Your equations use the cumulative probabilities, which is not correct in this situation. To clarify, the expected value is "the probability of observing an event between 0 and 10 seconds times the outcome, 10 (this is just the label for the any event that occurs between 0 and 10 seconds) + the probability of observing an event between 10 and 20 times the outcome, 20 (again this is just the label for any event that occurs between 10 and 20) + the probability of observing an event between 20 and 30 seconds time the outcome 30 + ....

[@waisyousofi9139]

AS USUAL, BEST!

[@statquest]

Thanks again!

[@AG-cx1ug]

What if the data is erratic and doesn't fit with any of the lambda values to make the curve?

[@statquest]

Then you can either use a different distribution, or you can try to approximate things with a histogram.

[@AG-cx1ug]

@@statquest Thank you!

[@alikhalili9961]

Where is the n-1??????????? I am outraged. Lol. By the way, great video. Looking forward for more.

[@statquest]

Unfortunately this is just the first of many steps. :(

[@pigritor]

Dear Sir, could you do video on Linear Mixed models and GEE? Maybe you could create a separate donation target with threshold which you need to achieve to make one? It would be extremely useful. It is long waited and hard topic with a lot of contradictory info.

[@statquest]

I'll keep that in mind.

[@taotaotan5671]

probability is fun : )

[@statquest]

BAM! :)

[@taotaotan5671]

​@@statquest I guess the next video will talk the variance of a random variable :)

[@davonraymond3274]

appreciate you

[@statquest]

Thank you!

[@aysan7513]

omg you saved my life

[@statquest]

bam!

[@megasun]

Number is not right around 8:25, the rectangle area should be approximate 0.3894. I was very surprised that the given rectangle area of 0.4 is bigger than the given integrated result of 0.39, because the rectangle area looks slightly smaller than the area under the curve ... still I love these videos. And of course, for the purpose of this lecture, having more digits here is distracting and not helpful.

[@statquest]

Sorry if my inconsistent rounding through you off.

[@AG-cx1ug]

At 14:30 isn't the anti derivative just the integral?

[@statquest]

yep

[@ANGGAKAHFI-kr4yy]

Sorry sir i dont get it when u use 0.05 for the lambda,

[@statquest]

What time point, minutes and seconds, are you asking about?

[@b1ueberrycheesecake]

BAM!!

[@statquest]

:)

[@ahmedabuali6768]

Dear Professor, I would like to ask you if you have a very good lecture note for this book, introduction to mathematical statistics Robert v. hogg, I am ready to pay for that, I like your methodology in presenting.

[@statquest]

Unfortunately I've never read that book.

[@ahmedabuali6768]

@@statquest Ok, I would like to take this chance again to ask you if possible to add topics about Bayesian statistics. Many thanks and I am still following you :)

[@statquest]

@@ahmedabuali6768 I'm working on those videos right now. They should be out soon.

[@ahmedabuali6768]

@@statquest wow, very good, attached also the pdf presentation please, I am ready to buy it. Go ahead, dear Prof.:)

[@auzaluis]

BAM!

[@statquest]

:)

[@tedchirvasiu]

5 years later: if you watched this video hoping to learn exactly why we divide by n-1, you are one step closer to understanding this mystery, but not quite there yet.

[@statquest]

Yep. Sorry it is taking so long.

[@Thedavidk]

Wow, that got complicated really quickly

[@statquest]

Did you first watch the expected values for discrete variables: kzbin.info/www/bejne/gX3WkGqYbLh-n5Y

[@RoyalYoutube_PRO]

Has the mystery of 'n-1' been resolved yet?

[@statquest]

Not yet. The best I can do is give you a link: online.stat.psu.edu/stat415/lesson/1/1.3

[@pibruks]

$$ #BAM!!!

[@statquest]

:)

[@tim40gabby25]

09:50 Eyeballing, 0.4 no way is twice the height of 0.2

[@statquest]

Rounding errors.

[@tim40gabby25]

@@statquest Appreciate the response, tho you went on a bit. Old UK duffer here, enjoying the ride :)

[@MrAzrai99]

Just realizing the significance of the calculus I've learnt long time ago ":"(

[@statquest]

bam!

[@MrAzrai99]

@@statquest Please make a video on multivariate normal distribution next🙇‍♂️

[@statquest]

@@MrAzrai99 I'll keep that in mind.

[@satyamraj2779]

you are too cute with your bams!

[@statquest]

:)

[@TJ-hs1qm]

ahh... how come that Likelyhood and Probability actually mean different things ? 🤷‍♂️

[@statquest]

See: kzbin.info/www/bejne/porbf4aLebh5fpY

[@nir199820]

I am beginning to think that the all "dividing by n-1" thing is a hoax :)

[@statquest]

:)

[@Sarah-hu4wt]

I like how I am smarter than StatSquatch

[@statquest]

bam! :)

[@neptunesbounty1786]

1 downvote by StatSquatch

[@statquest]

dang! :)

[@bockarieb]

Probability density function tutorials Ghana university

[@statquest]

?

[@timehealthfit1891]

This is a great video, want to be youtube friends?

[@statquest]

bam!

[@ruiyucong3125]

这集学不明白了，已晕

[@statquest]

Sorry. :( Did you watch this one first: kzbin.info/www/bejne/gX3WkGqYbLh-n5Y