Power Analysis, Clearly Explained!!!

[@statquest]

NOTE: This StatQuest was brought to you, in part, by a generous donation from TRIPLE BAM!!! members: M. Scola, N. Thomson, X. Liu, J. Lombana, A. Doss, A. Takeh, J. Butt. Thank you!!!! Support StatQuest by buying my book The StatQuest Illustrated Guide to Machine Learning or a Study Guide or Merch!!! statquest.org/statquest-store/

[@JimtheEvo]

This video makes me feel so happy. I've had a paper in review for a little while, the editor and one reviewer noticed in one of my figures (in which I had shown the raw data and CIs) that a single datum point was not in line with the rest (n=6 not great but better than most molecular biology papers!). They asked me to add an extra replicate to the experiment. I refused on the grounds of p-hacking and showed them a power analysis (+93% power including the variation of the point) and showed the point is a Grubbs outlier. I'm still waiting for the editor to get back to me ;)

[@statquest]

BAM! :)

[@raycyst-k9v]

What did the editor reply?

[@sockaccount8116]

@@raycyst-k9v Editor said: BAAM!

[@karolgilbertosolanosuarez9094]

So... What happened? Do you have your DOI for us to read? 🎉🥳🤩

[@JimtheEvo]

@@karolgilbertosolanosuarez9094 I won ;) gurney et al 2020 microbiology Combinatorial quorum sensing in Pseudomonas aeruginosa allows for novel cheating strategies

[@raycyst-k9v]

Man josh i am trying to finish your playlist . Before this i had tried khan academy, brandon foltz, few books and i failed horribly. P value was something i could never grasp . I think the issue was that everyone talked about what is p value but never explained HOW it is calculated. You really go in the depth of how and why. And this one video right here has blown my mind. It just doesnt explain power analysis but also how you estimate population mean and how p values work . It's 5 am right now and i am still studying .Youve really blown my mind!

[@statquest]

Hooray!!! I'm glad my videos are helpful. :)

[@raycyst-k9v]

@@statquest HUGE BAM!

[@mathiasmadsen1992]

You've freaking done it again. I can't belive how simple this explanation was compared to the lecture I got in school. Thank you very much!

[@statquest]

bam!

[@noedits5543]

the other teachers cannot explain the concepts clearly because their concepts are not clear in the first place LOL. thats where our Mr BAM Sir rocks :D

[@peterkilindberg675]

Wow! I'm very impressed by your very clear explanations and calculations in this video. It was very easy to understand and follow. Big thanks from a PhD-student in Sweden!

[@statquest]

Thank you very much! :)

[@sushanthraj3455]

I am so happy to see the growth rate of Josh Starmer's StatQuest. I still remember I was one of the few subscriber who joined your fan base when the subscribers where in thousands!. Great going happy learning

[@statquest]

Thank you very much! :)

[@efrensuarez4950]

One of the best channels out there to learn statistics.

[@statquest]

Thank you!

[@khoiavo]

I learned more (and laughed more) in these 16 minutes than a whole semester at med school. Thank you!

[@statquest]

bam! :)

[@tanishadorn6126]

Thank you! For my paper my lecturer said we don’t have to include a power analysis, so it wasn’t taught… so glad I watched this because I was struggling to justify my sample size and now it all makes so much more sense.

[@statquest]

Awesome!!! :)

[@vladfarias]

Your videos are absolutely amazing! It would be fantastic if you could create (if it doesn't already exist) a video covering the entire hypothesis testing process, including all the steps. That would involve determining the sample size and addressing the temptations encountered along the way until reaching the final result.

[@statquest]

Great suggestion!

[@uberdonkey9721]

I've done so much stats but akways had a difficulty understanding power analysis. This is very clear, and practical.

[@statquest]

Thanks!

[@SubDonkess]

Fantastic! I’m taking a statistical modeling class with machine learning and this particular topic just wasn’t sticking with me. As advertised, this was super clear and nailed home all the key points!

[@statquest]

Glad it was helpful!

[@JS-wv7vn]

Thanks I haven’t slept this good in weeks

[@statquest]

noted!

[@MrDrache7]

Josh in May 2020: "Imagine there is a virus" 2020: "Hold my beer"

[@statquest]

:)

[@faye8212]

Thanks Josh, you are the best at storytelling when explaining statistics.

[@statquest]

Wow, thanks!

[@taotaotan5671]

Simulation is really helpful to understand this.

[@YASHKUMARJAIN]

Totally loving the series

[@statquest]

Hooray! :)

[@suhacutis23]

Thank you for making learning more interesting. Double bammm for the knowledge and funny remarks

[@statquest]

Thank you! :)

[@rakeshdey3993]

You are a great teacher, Josh ! Thank you so much. For a long time, I had difficulty in understanding this concept. Now, it is crystal clear.. :-)

[@statquest]

Thanks!

[@janicealmojuela9068]

watching a night before my finals for STaT! thanks josh

[@statquest]

Good luck!!

[@d_b_]

At 14:09, several assumptions and parameters are presented on the screen, creating the impression of the existence of "power analysis hacking." Furthermore, the concept of double-layered probabilities, where there is an 80% likelihood of correctly rejecting the Null Hypothesis and a 5% chance of randomly obtaining results below the threshold, is enough to make one's head spin. Nevertheless, this video provides the most exceptional and clear explanation.

[@statquest]

Thanks!

[@apoorvasrini2196]

This explanation is amazing!!! Thank you so very much!!

[@statquest]

Thank you!

[@PrabhakarKrishnamurthyprof]

Thank you Professor. I will use it for my class. It is so well explained, I learnt how to explain complex concepts.

[@statquest]

Glad it was helpful!

[@alexrinconp]

Dude, you have awesome explanatory superPower!

[@statquest]

Thanks! 😃

[@Samurai_Jack__]

I'm happy my fate made me study statistics after you made this playlist , Thank you so much for this amazing explanation

[@statquest]

Happy to help!

[@konstantinlevin8651]

Hey! After chatting for an hour with chatgpt about p-values, p-hacking and power analysis, I wanted to test my knowledge with a simple case and chatgpt suggested me a coffee shop experiment. The experiment is going to be about a new bean and if it improved the customer satisfaction or not. I'll try to prepare a dataset for this. If I can, I'll try to share! Also, thanks a lot for the content. I want to be a machine learning engineer and decided to learn the fundementals of statistics for that, but statistics basically changed the way I see the world ahahahah.

[@konstantinlevin8651]

I have to learn how to do those statistical tests though :)) I'll watch the quest

[@statquest]

bam!

[@bowenzhang8714]

This tutorial is great and make a lot of common sense.

[@statquest]

Thank you!

[@ericzheng4815]

The interesting examples really gave me a new perspective on the problem. Appreciate this! Thank you so much!

[@statquest]

Glad it was helpful!

[@abdelghanyaref6516]

i swear to god that you are a life saver man

[@statquest]

:)

[@nikhilgoyal204]

Josh, thanks so much for your videos - so clearly explained relative to other content I've seen on inference. I do have a few questions. 1. In an A/B testing scenario, we don't know the mean and the standard deviation of the distribution of the treatment group before hand. We do know the mean and the standard deviation of the prior distribution. If I want to estimate sample size required for different effect sizes, holding the chosen p-value threshold constant at 0.05 and power at 0.8, how would we do that given that we don't have s2 or standard deviation of the second group? 2. Also since we don't know the mean of the treatment group's distribution, how would we calculate the estimated difference in means to plug it into the formula for effect size? 3. Do you have a video on How to pick the right test?

[@statquest]

1) Do a pilot study to get a sense of the mean and standard dev. 2) Same 3) Not yet.

[@chronisfatouros9906]

Fantastic explanation, very useful and awesome graphics, high educational value!! CLAP CLAP

[@statquest]

Thank you very much! :)

[@jiayinruan9282]

so clear and understandable !!!! Thank you very much!!!

[@statquest]

Glad it helped!

[@KnowDAOself]

excellent video series in general, and particularly good video on Power Analysis! Thank you for making and sharing!

[@statquest]

Thank you!

[@pandapanda7966]

i have a question. so knowing statistical power, significance level, effect size, and sample size are all related through power analysis, then wouldn't choosing a statistical power (or even significance level, as we know 0.05 threshold is just for teaching demonstration and in reality it can be any different amount) also be consider p-hacking by effectively getting a desire sample size in "remote" in order to have it prove (or disprove) hypothesis at the researcher's discretion? or is there an objective way to determine the statistical power (and significance level)? most of the materials i read often says "commonly use power = 0.8 or alpha = 0.05" or even heard "amount chosen at researcher's discretion" but without giving sufficient reason for picking those amount.

[@statquest]

There are a couple of things to say. 1) Because we select all of the thresholds (power, significance etc.) before doing the experiment, we end up with a sample size that should not be biased in terms of having a higher probability of giving us a false positive of false negative. The key is that we set those parameters before we do the experiment and then we have to live with the result no matter what it says. If we didn't get the result we wanted, and then we did it again, then that would be p-hacking. 2) The thresholds for significance and power are often "field specific" and depend on how easy or difficult it is to control things. In physics labs, they can control things relatively well, so they tend require stricter thresholds. However, when working with human data, where it's hard to control anything, then they tend to have more relaxed criteria. The thresholds also change depending on how serious the consequences are of getting things wrong. For example, if we are testing for a Ebola, we might want to error on the side of caution and allow more false positives so that we can minimize the false negatives.

[@nishat_zaman]

This lecture was really excellent 😊

[@statquest]

Glad you liked it!

[@MrCracou]

This is, as usual, excellent. I'm a little bit surprised that you do not show alpha and beta for a given test (as the surface below or above the critical value). It allows people to understand the main problem of the power analysis: you need to bet on the effect size as, by definition, it's unknown. It leads, as you know, to soo many problems as we alsways discover too late that the actual mean difference is smaller than expected (also known as the "curse of just a little bit too small sample". The most common solution being a ritual sacrifice of a random slave. Sorry, I want to mean a PhD student.

[@financequant]

Excellent lesson on power analysis. I am immensely impressed by StatQuest’s instructional videos. One criticism here, however, is that the final computation of the necessary sample size to achieve the desired power could have been covered rather than telling a student to find an app on some university’s web site. I think most students could probably work out the math but it seems to go against the overall grain of the talk which careful walks the student through a well chosen toy example that nevertheless fully elucidates the process.

[@statquest]

Thanks! I decided to not go over the math for calculating power because I did not want to give the illusion that there was a single equation that fit all situations. In reality, for each test, there is a different equation that we need to use to calculate power and, rather than worry about all of that, I thought it would be easier (and much more practical) to show how power analyses are done.

[@drrajkumarsrihari]

Thanks

[@statquest]

Wow!!! THANK YOU VERY MUCH!!!!

[@TheodoraPapadopoulou-c3j]

Thank you, very helpful! I am a new bioinformatician and planning to do differential gene expression with DESeq2 comparing a human sample size=1000. How could we apply power analysis for RNA seq differential expression?

[@statquest]

I believe there are some standard RNS-seq power calculators out there that you can use. I remember using one, but that was 6 years ago and I bet there is something better now.

[@ArinaNotHarina]

Awesome channel. Thank you for what you do

[@statquest]

Glad you enjoy it!

[@shrikantdeshmukh7951]

Gpower is very good software for sample size calculation.

[@Niilamaru]

Nice way to start a remote workday.

[@drpkmath12345]

Niila Saarinen yes right?

[@statquest]

Hooray! :)

[@pradeepkumar-ew1ze]

Hope your boss is not the subscriber :P

[@ByeolKim-w8q]

I love it when you're saying "BAM!" lol

[@statquest]

BAM! :)

[@sanaaharrass6239]

great explanation. Thanks!

[@statquest]

Thanks!

[@ZHL-n9l]

Thank you Josh for being so careful and patient. I would like to ask you a question that I am not quite clear about. You used s when calculating pooled estimated standard deviations and represented s with a dotted line on the normal distribution. However, it seems that s is very wide in the figure, which looks like 2s. I want to make sure that this s is what we call s, right?

[@statquest]

's' looks pretty accurate to me. About 65% of the area under each curve should be under the dotted line.

[@suk-yongjang8885]

Excellent explanation...

[@statquest]

Thank you! :)

[@Quijanos1]

Great explanation! Thank you.

[@statquest]

Thanks!

[@zahrarezazadeh293]

Hi Josh, thank you a lot for the awesome videos! I have two basic questions: 1. The denominator for the pooled estimated SD in a general condition is the number of the distributions (and not always 2), right? 2. What is the deal with statistics power calculator? isn't there simple formulas we can use ourselves? and does the googling we do to choose one, involve the nature of our experiment, or we just randomly choose one?

[@statquest]

1) I'm not sure how you could have more than two distributions, but, presumably, if you did, then you would divide by that number. 2) Every single statistical test has a different formula for doing power calculations. And there are a lot of statistical tests, because there are a lot of different experiment types and data types. Going through every single statistical test would take forever and, to be honest, be pretty boring. So the best thing to do is to google "power calculator" and you'll find a page that has a pulldown menu where you select the test that you want to do (t-test? chi-square goodness of fit? K-S test? etc.) and then plug in your numbers. bam.

[@davidalfonsoriveraruiz9077]

Excelent video, thank you very much.

[@statquest]

Glad you liked it!

[@lunapopo8415]

15:04 It seems tricky to find the right df for t score, for such small sample calculation, when n is unknown. Is it a trial and error process to find the right df?

[@statquest]

df for a t-test is directly related to the sample size. So if you can solve for 'n', you can solve for the degrees of freedom.

[@ShivamTiwari-on2kl]

You're the best, boss!

[@statquest]

Wow, thanks!

[@jingjingli101]

followed you on bilibili haha, cant wait to your new vedios, so I came here :)

[@statquest]

bam! :)

[@gayanbandara1909]

Supereb sir... Thank you..

[@statquest]

Thank you!

[@lifewithrahman7955]

It was very very very easy explanation

[@statquest]

Thanks!

[@gabrielagalindo6339]

Thank you, great explanation

[@statquest]

Thanks!

[@aashraysaini994]

Thank you Josh for all your videos! they help make statistics look less deadly. Would look forward to some content on bayesian statistics and then maybe solving problems using both bayesian and frequentist methods :)

[@statquest]

Will do!

[@haneulkim4902]

Thanks Josh! Just want to make sure, the green and red normal distributions are created from 1. collecting data_size=n from one group, calculate mean 2. repeat 1 R times 3. use R means to create green/red normal distribution correct?

[@statquest]

Unfortunately that's not correct. The green and red distributions are the "population" distributions that we are trying to estimate with relatively small sample sizes. For details on population distributions and how they are different from samples, see: kzbin.info/www/bejne/rJrOnJytn7aknLc and kzbin.info/www/bejne/iau9Z3qmmMuih7s

[@bnurpainting]

As easy as it can be! Thanks...

[@statquest]

You're welcome!

[@SergeySenigov]

Josh, in terms of the statement 7:49 «because we don’t have a lot of confidence in the estimated means, we’ll end up with a relatively large p-value», can we conclude statement like this: we’ll end up with relatively more both large and smaller p-values, ie p-value more «unstable»?

[@statquest]

That sounds like a reasonable statement. However, how "stable" or "unstable" the p-value is probably depends on the underlying distributions.

[@hussienali6561]

Thanks for your illustration. but to understand it more, you explained it on an example of two sample test. What if we run one sample test ?

[@statquest]

It's the same - even with just one sample, the sample has a distribution that may or may not be the same as an ideal distribution that we are comparing it to.

[@jmyable4]

Top-notch! Thank you!!!!!!

[@statquest]

Thanks!

[@Dragonite789]

Thanks Josh. I wanted to ask what should you do if all your data comes from one giant experiment/repository. Your explanations usually assume we have done multiple experiments of small sample size and can generate pooled statistics. But if you did one big experiment with a very large sample size, should you artificially separate individual test into small groups so you can generate pooled statistics? Thanks again, I hope i make sense.

[@statquest]

Most of the time people do one big experiment and collect all of the data at once. This is the standard procedure. However, in the video, I show what happens if you can repeat that same experiment a bunch of times to give you a sense of how whatever you are estimating (like the mean) has less variation when you have more samples. Thus, the repeated experiments in this video are just an illustration to give you confidence that a large sample size will be more accurate that a small one.

[@andrewl372]

Thank you for the great video! I have one question though- 1:51 In the normal distribution graph, If the variable is day to recovery which can't be lower than 0, then would the estimated populatiln probability distribution be following poisson distribution..? Sorry if I may be confused with the concepts in advance!

[@statquest]

Believe it or not, the normal distribution is often used to approximate the poisson distribution. The idea is that if the means are far enough from 0, then it's a good fit. This is due to the central limit theorem. For details, see: online.stat.psu.edu/stat414/lesson/28/28.2

[@andrewl372]

@@statquest I understand the explanation, thank you!!

[@mikaelfiil3733]

Nice videos Josh. One thing concerning p-values that I think should be mentioned is, that they are a continues value, so 0.05 is somewhat arbitrary. In terms of p-hacking and the notion of correlation vs. causality, taking 0.05 too seriously as evidence can be troublesome. You may actually miss out on some good science, just because the p-value was > 0.05 and of course, you may be too optimistic in the other direction. Maybe you already mentioned this, and I just missed it :-)

[@statquest]

I mention this in my video that explains p-values: kzbin.info/www/bejne/rJbQi6d7gptmfbs

[@hexu8043]

every time when I watch this series it reminds me of Moss in the IT Crowd :D

[@statquest]

Nice!

[@alexw31415]

Been following the Statistics Fundamentals playlist quite well up until this point. Now things are getting a bit meta. P-values are the threshold for false positives, to be able to say "we have a 5% tolerance for getting the wrong result" - that's already pretty meta. Then FDR and the BH-method look at "distributions of p-values" to sub-divide the true positives from the false positives. Double-meta. Now on top of that we are doing a "power analysis" to get an 80% chance of trusting our p-values? Triple meta? We already set our tolerance of 0.05, saying "we know we'll get it wrong 5% of the time" so why do we then care about if those 5% are true/false or "close enough". I'll keep watching but FYI that's the thought process for a new viewer.

[@statquest]

Power is sort of the opposite of a p-value. p-values tell us about the probabilities of false positives. Power tells us about the probabilities of false negatives.

[@alexw31415]

Very nice and simplified way to put it, thank you Josh! Keep up the good work

[@suhailahmad4809]

What a way to explain the things😍

[@statquest]

Thank you! :)

[@kairoselvarro6546]

Thank you for the detailed explanation! But...I am having trouble finding a reliable power calculator online that allows me to input the Effect Size...perhaps you could recommend a few? What power calculator did you use to get the value of sample size 9? Regardless, I loved the first part of your video! Very detailed, very interactive, thank you for this!

[@statquest]

www.statskingdom.com/32test_power_t_z.html

[@danlabrador]

14:55 If the calculator suggests a sample size of 9, is it safe to increase the sample size or should I just stick to the calculated sample size?

[@statquest]

You can always collect and use more measurements than the minimum.

[@abolimalwadkar5216]

Thanks for this video I now get the 'effect size' true meaning I am unable to find the sample size calculator. I am learning stats to work on SPSS but was curious how you got the value 9 for the example

[@statquest]

Just google "sample size calculator" and you should be good to go.

[@skytoin]

Hi, there are different power calculators online and they require different inputs. How do you call the one that needs: power, threshold, effect size?

[@statquest]

You have to know what kind of test you want to use - once you figure that out, the inputs will all make sense.

[@jourdanbrune4856]

What if I am trying to compare three groups? How would I calculate the effect size? What other key words should I google in addition to "statistics power calculator"? Thanks in advance for answer my questions, your videos are awesome!

[@statquest]

Add "ANOVA" to your search (which compares 3 or more groups). In general, if you have a specific test in mind, just add that to your search.

[@a45701]

According to the video, calculating the effect size in the end using the difference of means, it is divided by the pooled estimated standard deviations. What standard deviation (std) is that? The std of samples, or the standard deviation of the estimated difference in means? (the std of the mean should be 1/sqrt(n) of the samples std) Does the effect size follow the same calculation as Cohen's d ?

[@statquest]

It's the standard deviation of the samples.

[@rohitbansal3032]

You are a Genius! Super

[@statquest]

Wow, thanks!

[@ayesha-w3w4f]

Hello, thank you so much for this video. I have a question as I want to try do a power analysis for my data however it has a very small sample size so I could not assume that the data was normally distributed. If you do do the Shapiro Wilks test most of the data appears to be normally distributed. In the actual study I compared medians and IQRs rather than means and SDs. Does this mean I cannot do a power calculation because I decided to use non-parametric tests for my data because I couldn't assume normality?

[@statquest]

There are plenty of power calculators out there for non-parametric tests.

[@raffaelepiccini3405]

Great video, I’m still a bit confused on how having different sample size for the two groups would affect the calculations

[@statquest]

Here's something that might help: stats.stackexchange.com/questions/108079/can-i-do-a-t-test-power-analysis-for-unequal-size-groups-which-produces-2-differ

[@raffaelepiccini3405]

@@statquest thanks!! that was a useful link

[@titustan2700]

I have a question: if the original data that are used for the sd and mean of the two distributions are not representative due to small sample size, does all of this power analysis breaks down. That is to say, power analysis is only based on previous large sample size data?

[@statquest]

No, it doesn't break down because we give ourselves the potential to be wrong. When the "power" is set to 80%, that means we only have an 80% chance of correctly rejecting the null hypothesis. If we want a 90% chance of correctly rejecting the null, we need to increase the sample size.

[@arthurribeirodeabreuchaves1131]

Great, easy, and simple video. Could you provide a reference for the formula you used to calculate Cohen's (d)? Thanks!

[@statquest]

en.wikipedia.org/wiki/Effect_size NOTE: The pooled standard deviation is simplified to assume the same number of measurements from both categories.

[@harveyhensley875]

Wonderful video, very clear as always. Often the terms 'type 1 error' and 'type 2 error' are used in these discussions, or α and β. Was there a reason for avoiding this terminology, aside from just generally avoiding jargon?

[@statquest]

I really don't like that terminology. I'd rather people say "false positive" or "false negative" instead of "type 1" or "type 2".

[@martinotanasini3716]

Thanks for the great video! Do I understand correctly that the sample size determined from the power analysis depends on the means and variances estimated from the first experiment? How do I deal with the fact that this first experiment could result in estimating means and variances far away from the population's?

[@statquest]

You have to start with some general sense of the variation in the data. It doesn't have to be perfect, but it's the best you can do.

[@josephgan1262]

Hi Josh! Thanks for your amazing explanation! Appreciate greatly! I have a questions. I read from a book, it says that large sample size can have unwanted implication. QUOTE ....'By “too high” we mean that by increasing sample size, smaller and smaller effects (e.g., correlations) will be found to be statistically significant. The researcher must always be aware that sample size can affect the statistical test either by making it insensitive (at small sample sizes) or overly sensitive (at very large sample sizes). UNQUOTE So does it mean that even the test we are conducting is from the same distribution(or null hypothesis is true) . It will some how reject the null hypothesis given large enough sample size? Thanks!

[@statquest]

Regardless of the sample size, you always need to make sure that the effect size is large enough to have "meaning". For example, if taking an expensive medication increases life span by 2 seconds, then who cares if it is statistically significant or not - 2 seconds is not worth taking an expensive medication. So when ever you do statistics, don't just look at the p-value, also look at the effect size and make sure it is meaningful.

[@SenthilKumarSubramanian-x6d]

Hi, very useful material and well explained. Thanks. What should be the power if i want to accept null hypothesis? How should be the overlap and sample size, in this case?

[@statquest]

Unfortunately, traditional statistics is not designed to accept the null. The best we can do is to "fail to reject the null". You can make this failure relatively convincing if you select a relatively high power (like 95%) for rejecting the null and still fail.

[@SenthilKumarSubramanian-x6d]

@@statquest I have a subject interviewed at time 0 and at the same time, the subject has given rating for those questions asked via an app. Now, such measurements are repeated at an interval. I want to show that there is no statistical difference between interview rating and self-rating via app. I need to find out the sample number for this analysis. What should be the power , alpha and effect size to find out the sample number? I appreciate your help. Thank you

[@nelhuens7590]

Great explanation! What do you do if you want to compare a normal distribution with a non-normal distribution?

[@statquest]

If the sample size is large enough, the distribution doesn't matter (per the Central Limit Theorem: kzbin.info/www/bejne/j3LPe3Z7ea1lq7s ). Alternatively you can use non-parametric methods. These do not depend on the distribution.

[@nelhuens7590]

@@statquest Thank you very much! We have a really small sample size, so I don't think that is an option. Do you maybe have a link to wich non-parametric methods we can use?

[@statquest]

@@nelhuens7590 Unfortunately, I don't. But if your data is continuous, instead of using a t-test, you can use a Mann-Whitney U test: en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test

[@deepakmehta1813]

Hi Josh, Thoroughly enjoyed the video. One quick question though: if the two means are d1 and d2, then how do you to take the difference between them, should it be |d1-d2| or one can used either d1-d2 or d2-d1. On a side note my kids have started complaining - They say that I am always watching stat quest :)

[@statquest]

The order doesn't matter (and we don't use the absolute value either). If you want to learn a very cool way to compare two means, check out my playlist on linear regression. Believe it or not, a test to compare means is a type of linear regression: kzbin.info/aero/PLblh5JKOoLUIzaEkCLIUxQFjPIlapw8nU (Tell your kids that I apologize for making your watch more StatQuest! :)

[@syhusada1130]

So if power analysis told me to have 500 sample, and I have only 5, can I reach that 500 with bootstrap 100 times?

[@statquest]

First of all, when we do bootstrapping, we usually do at least 1,000 and often 10,000 or more permutations. And, unfortunately, these permutations do not increase our power. All they do is give us a sense of the the distribution of whatever metric we are calculating (usually the mean) given the original sample size. Thus, if your sample size is 5 and we use that to calculate the mean, bootstrapping would then tell us how a mean, given n=5, is distributed. It doesn't increase the probability we will correctly reject the null hypothesis.

[@jakubmazur1159]

Love your videos, thanks! :) Quick question though - from this video it seems that we should google 'statistics power calculator' after we set up 3 things - alpha, power and effect size. We can easily choose alpha and power, but what about effect size? It seems to be the thing that you are able to calculate after the test finishes, you can't set it up beforehand. Or maybe you meant something like 'minimal effect size that is enough', e.g. from business perspective?

[@statquest]

It depends on the area you are working in. For example, in biology, we often want a 2-fold effect size. In physics, a smaller effect size might be good enough. So you have to know what people want in the field to determine the value.

[@jakubmazur1159]

@@statquest thank you for the answer. And in case of AB testing? It would be nice to know beforehand what would be the sufficient sample size. Then what should we set up in place of effect size?

[@statquest]

@@jakubmazur1159 Again, this is domain specific.

[@Mostafaseyyedabadi]

Thanks for the great video. In your example you said 9 measurements per a group is enough. What does group mean in correlation studies? Do we have more than one group in these studies?

[@statquest]

What time point in the video, minutes and seconds, are you asking about?

[@Mostafaseyyedabadi]

@@statquest thanks for your quick reply. I asked about minute: 14 and second: 59

[@statquest]

@@Mostafaseyyedabadi If you're just trying to establish a correlation, you just have one group. You can determine the sample size using a sample size calculator for regression. For example, you could try this one: www.statskingdom.com/sample_size_regression.html

[@Ginimo]

Hi Josh, for a trustworthy result I would combine Power Analysis to determine the sample size and after that calculate P-values with the FDR correction. Is this right? Or is the FDR correction not more needed after Power Analysis? Many thanks for your response!

[@statquest]

If you do multiple tests, you should always use FDR.

[@matheusfuji]

Excellent!!

[@statquest]

Thank you!

[@simkort5799]

Great series of video again Josh!! I have one question tho, and I have tried to ask Chatgpt, and it didnt help much... Given we rejected the null, if we say alpha is the possibilities of falsely reject the null, lets say it is 0.05. And then the opposite would be correctly reject the null which is 1-0.05 = 0.95. Isn't this exactly the definition of Power? So given alpha = 0.05, the chance of correctly reject the null is 95%, and that is our power? I know it is somehow false, but i cannot figure out why

[@statquest]

I actually have an entire video that answers your question: kzbin.info/www/bejne/iKTGZq2krLdofKM (I'll give you a hint: 1-alpha is not what you think it is. 1-alpha is related to the probability that we will fail to reject the null).

[@simkort5799]

@@statquest Thanks a lot Josh! I realized I made that mistake haha! 1-alpha is consist of two parts, 1.correctly accept the true null 2. falsely accept the false null and Power is the chance of correctly reject the false null. So they are not completely the same

[@ShuaiWang12]

Thank for making these videos. One question on the beginning of this video. What is the actual method behind to "do a statistical test and compare the means and get a p-value=0.06"? From what I read, is T-test for two sample mean in this case? Thanks!

[@statquest]

Yes, in this case I'm using a t-test.

[@vladfarias]

Man, I'm totally hooked on your playlist (it deserves to be on Netflix!). When you mentioned having a sample size of 9, I got a bit confused. How many samples, each with a size of 9, should I collect? I always struggle with this. For instance, would it be the same if I had 3 samples, each with a size of 3? Could you clarify that?"

[@statquest]

What time point, minutes and seconds, in the video are you referring to?

[@vladfarias]

@@statquest 14:56. Anyway, would it be the same? (One sample size 9 and 3 samples size 3)

[@statquest]

@@vladfarias It depends. For example, 1 person getting 9 measurements could be different than 3 people getting 3 measurements each because there could be variation in how the 3 different people collected measurements. Or if we use the same person, but collected 3 measurements on 3 different days, because there could be extra variation due to the different days. So your estimate of variation needs to take how the data will be collected into account.

[@Zombiezzz101]

So just to be clear. At the end of the video it's shown that we need a sample size of 9. If we go over the sample size of 9, we may run into p-hacking. If we go under 9, we will be less confident in our results thus our power decreases. So my question is what is worse, obtaining a lower sample size or obtaining a higher sample size than what is recommended? I've always assumed that obtaining the largest sample size possible is the way to go since it will give you a closer estimate of the true population mean. Is this not the case? Is there always some "goldilocks" sample size number that should always be computed before running a hypothesis test? Thanks

[@statquest]

You can never have too much data. Collecting more than 9 samples doesn't amount to p-hacking. p-hacking occurs when you do the same test a bunch of different times and then cherry pick which one we want to believe. In contrast, if we have a large data set and we do a single test with all of the data at one time, then we can only have more power to detect differences if they exist.

[@baksteen2420]

Thanks for another great video Josh! There's one thing I still fail to understand, though: Isn't it true that a p-value and Power measure the same thing, namely the chance of (correctly) rejecting the H0? And if this is the case, is a chance of 80% of correctly rejecting the H0 not a downgrade from having a 95% chance when using an alpha of .05?

[@statquest]

No, p-values and power are fundamentally different. p-values assume that the null hypothesis is correct. Power assumes that the null hypothesis is not correct. This makes pretty much everything about them different.

[@baksteen2420]

​@@statquest thanks a lot for replying! I clearly still have some work to do😁

[@statquest]

@@baksteen2420 To learn more about p-values, try: kzbin.info/www/bejne/rJbQi6d7gptmfbs and kzbin.info/www/bejne/gILGZKyuZZKEb6c

[@lupita3689]

Wait, but you assumed you already knew the two distributions? Since you'd need the population mean and s.d. to calculate the effect size?

[@statquest]

The mean and standard deviations are estimated from the data.

[@lupita3689]

StatQuest with Josh Starmer But if you already have the sample mean and data, wouldn’t that mean that you already finished your experiment and have the sample size fixed? To me this seems like a chicken and egg problem. Or are we talking about an iterative process where you finish the experiment, check if the sample size is sufficient for the Power you want, then do more experiment to get more sample?

[@statquest]

You can estimate the mean and standard deviation from a preliminary experiment, or from a literature search, or just use an educated guess. I mention all of these options at 13:57

[@lupita3689]

StatQuest with Josh Starmer Ah, sorry I missed that, and thanks!

[@av3499]

@@lupita3689 that was exactly my thinking as well. 1. To check if samples belong to different groups, collect samples. 2. To know how many samples to collect, use power formula. 3. To get power formula, plug in mean & SD of the 2 different groups.

[@datamanager-th]

Thank you for the excellent video. If you please, I would like to ask you a simple question. In order to calculate the effect size, you need to know S which depends on the sample size (in the video it is 10). Then you use this to estimate the "sample size" which is 9. What happens if they are so much different? Can this happen in 2 cases 1. the start sample size >> the calculated one and vice versa.

[@statquest]

In theory, 's' (the standard deviation) does not actually depend on the sample size. We can get it from any source or just guess based on previous experience. Thus, if the output sample size is very different, that's OK.

[@datamanager-th]

@@statquest Thank you very much for your answer.

[@maximillianweil2672]

Is it possible to calculate the sample size required even if the distributions can't be assumed to be normal distribution? I would think it is possible by using the Mann-Whitney U test but can't seem to figure out how. Thanks for the great explanation anyways!

[@statquest]

You can definitely calculate power for a mann-whitney U test. www.benchmarksixsigma.com/calculators/sample-size-calculator-for-mann-whitney-test/

[@niazsayakhan1791]

Thank you Josh! I have a question. Could you collect a sample from Drug A and Drug B and use the two samples to calculate an Estimated Mean and Standard Deviation in order to calculate the Sample Size required for future research?

[@statquest]

Yep!

[@niazsayakhan1791]

@@statquest BAM!!! Thanks, Josh!

[@ABeardedDad]

Awesome video! There's really nothing like being able to visualise a concept. Is there a reason you never mentioned the standard error when you were explaining how larger sample sizes increase the confidence that your sample mean is close to the population mean? I mean standard error is literally the measure of that confidence right?

[@statquest]

Yes, the standard error is the measure of that confidence. However, I omitted it because I was simply trying to convey the concept of power, rather than dive into the technical terminology.

[@casualcasual1234]

Thanks for the video :) and I got 2 questions: 1. If I cannot find any published data for the effect size, what should be the sample size to generate a preliminary data? 2. is the threshold of significance (alpha) = p value?

[@statquest]

1) Even if there is no published data, there is probably a standard sample size used within your field. Or just take a guess. 2) Yes