Sample Size Estimation in A/B Tests Explained!

This video is a step-by-step guide on how to estimate sample sizes for A/B tests. This is a very important concept to master in order to ace your Data Science interviews.
Everything You Need to Know About A/B Testing in Data Science Interviews
• Crack A/B Testing Prob...
🟢Get all my free data science interview resources
www.emmading.com/resources
🟡 Product Case Interview Cheatsheet www.emmading.com/product-case...
🟠 Statistics Interview Cheatsheet www.emmading.com/statistics-i...
🟣 Behavioral Interview Cheatsheet www.emmading.com/behavioral-i...
🔵 Data Science Resume Checklist www.emmading.com/data-science...
✅ We work with Experienced Data Scientists to help them land their next dream jobs. Apply now: www.emmading.com/coaching
// Comment
Got any questions? Something to add?
Write a comment below to chat.
// Let's connect on LinkedIn:
/ emmading001

[@dwardster]

Hi two questions: 1. I see this formula is used for two-sample T-tests, but what about the more common case of a two-sample Z-test for proportions? What would we use to calculate sample size in an interview in that case? 2. How do we estimate σ before running the experiment? Just take a sample before running the experiment and use its standard deviation to estimate σ?

[@vitheltone]

Super insightful, but please, at min 1:30 it says "acceptance of H0" , we never accept H0, its reject or fail to reject :)

[@waynewang2071]

The math trick was wrong: x = \Phi(z_x) for 0 < x < 1. But you can still use the fact that z_{\beta} = - z_{1-\beta} in this case, which will give you the correct formula in the end.

[@sitongchen6688]

Hi Emma, thanks for your great video! One question, is here I know you are assuming control and treatment groups have equal variances. But in reality, they may not have same variance, therefore when applying this formula, which variance shall we pick or shall we use pooled variance? Or shall we ensure that those two groups have the same variance? Thanks a lot!

[@user-mq9si8eh5k]

Wow, so much information, thanks a lot!

[@hiteshgupta7099]

Hi Emma, thanks for this video. Is the sample size formula to calc only if there are 2 variant groups(control + treatment)? If so how will the formula change if we need more than 2 variant groups?

[@shreyaschaturvedi1933]

hi emma, can you explain how to calculate the standard deviation in this formula? since the control and treatment group will have different variation, should we use some kind of pooled/unpooled standard error for this number?

[@songxiyou2347]

Hi Emma, Question : beta = P(accept H0 | H0 is false) is a conditional probability, why we say beta = P(accept H0) directly?

[@jingcaomath]

same question here? Everyone can explain this?

[@weichiyao5712]

I think this is a typo or something. Formally, Beta = P(Do not accept H0 | H1) = P( |xbar / s /√2n| < z_{alpha/2} | H1). Conditioning on H1 means xbar ~ N(µc-µt, s /√2n). That's why in the next step she subtract µc-µt from xbar.

[@chihiroa1045]

Thank you.

[@chelseazhang1458]

Hey Emma! Your videos are very helpful and efficient for preparing interviews! Can you explain a little bit on when should we use t-test vs z-test or what's the difference between the two? I'm a little bit confused about the transition between the two. Thanks!!

[@emma_ding]

More videos about statistic is upcoming. Stay tuned!

[@xinyuntang8124]

T test is for comparing two samples and z test is for comparing sample and population, based on your use case, it’s usually pretty obvious whether you should use t test or z test

[@AdiJ8]

z-test assumes a population standard deviation/ t-test assumes you don't know the population std and so you estimate it with a sample std

[@owenho561]

Your video is very helpful. One question for clarification when you mention delta square is the difference between control and treatment. What the variable is difference of? Can you give us an example. The variance is clear. Thx in advance

[@lucasdeoliveirasilva8466]

Is the n the sum of control and variant group sizes? i mean, should i consider n=16*(sigmaˆ2)/(lift ˆ2)= n(control) + n(variant) ? (assuming the AB test is 50/50) Or n=n(variant)=n(control)?

[@yuchingyang3089]

Hi Emma, Thanks for your clearly explained for sample size. I would like to know that in practice, how we can determine the minimum detective effect (delta), is it the same as the practical significance boundary?

[@hiteshgupta9286]

usually yes.

[@owenho561]

This is super helpful! Thanks for sharing. One quick clarification on delta square. Is the Uc and Ut population delta between control and treatment group or it is something else? Can you give us an example?

[@emma_ding]

mu_c and mu_t are population means of the control and the treatment groups. Those are unknown and estimated from the data.

[@joe162840]

There is something wrong here....at 1:49, shouldn't it be accepting H(1) or rejecting H(0)?

[@zdaman011]

Great explanation! Out of curiosity, when would we need to apply a formula like this? Wouldn't we already know what the sample size is if we have a sample variance?

[@zdaman011]

@@Doorknob985 That makes sense, but what's confusing me is that the sample size formula requires us to have the variance of the sample means already, which implies we've already ran an experiment

[@emma_ding]

@@zdaman011 No, sample variance can be obtained from the data before the experiment (it's the same as the variance in the control group after you run the test), exactly as Ravi mentioned above i.e. from existing metric. Sample size is the size of the data thats needed to get a certain power of a test. This is an over-simplified formula as it assumes sample variance is the same in control and treatment groups, but it would be helpful for interviews. In reality, it's more common to have a "ramping" process to control risks rather than splitting all users into either control or treatment.

[@learnrepeatlearn]

@@emma_ding Hi Emma! Thank you for the great video. I have a quick question: What happens if the variance is different between control and treatment, is there another equation we can use to calculate the sample sizes for each? I am assuming that control and treatment group may not necessarily have the same sizes here

[@popo-je8ze]

I found that some practical issues are quite hard 1. how to calculate required sample size for ratio or quantile metric ? 2. if we already know that there are some correlations between our sample,thus,underestimating the variance, how to estimate required sample size correctly either for AA test or AB test

[@rickyg3390]

Hi Emma, isn't phi(Z(x)) = x? instead of -x?

[@martingai7333]

I agree, this is also where I got stuck. Have u figure it out?

[@jianzezhou1301]

This part is right. Actually it is the last part to be wrong, Z0.025 = 1.96 and Z0.2 = 0.84.

[@kagelsmith997]

At 1:04, why is the variance 2*sigma^2 / n? Why do you need to multiply by 2?

[@davidlee5048]

Because the pool variance is sample variance+treatment variance, and we assume control and treatment's variance are equal

[@jianzezhou1301]

The last part is wrong, Z0.025 = 1.96 and Z0.2 = 0.84. Actually if Z0.025 is negative everything after the absolute value part is wrong.

[@EvanZamir]

Is there a book that contains this derivation?

[@emma_ding]

hmm, not I'm aware of. But there're online resources on it if you search on Google.

[@mvijayvenkatesh]

I am using "Applied statistics & probability for engineers" by Douglas Montgomery and George Runger, 2nd Edition. Chapters 7,8,9 deal with these topics and the book is an excellent resource.

[@overseasafrican9899]

@evan zamir, You can find another derivation of this formula in Gerald van Belle's book Statistical Rules of Thumb. Chapter 2, page 30. van Belle has a free pdf of this particular chapter posted on his web page.

[@adrianusqueiroz]

How can we have sample variance, if we did not perform the experiment yet? this is making me confused!

[@lucasdeoliveirasilva8466]

that would be an estimate...i think

[@empiricalformulas9693]

right answer, incorrect alpha and beta.