A/B Testing Fundamentals: What Every Data Scientist Needs to Know!

Data Scientists Must Know | Data Science Interview Must Watch: A/B Testing Fundamentals. I will dive deep into practical problems in A/B testing and share things I learned from the book Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing.
🟢Get all my free data science interview resources
www.emmading.com/resources
🟡 Product Case Interview Cheatsheet www.emmading.com/product-case...
🟠 Statistics Interview Cheatsheet www.emmading.com/statistics-i...
🟣 Behavioral Interview Cheatsheet www.emmading.com/behavioral-i...
🔵 Data Science Resume Checklist www.emmading.com/data-science...
✅ We work with Experienced Data Scientists to help them land their next dream jobs. Apply now: www.emmading.com/coaching
✔️ Cracking Hypothesis Testing Problems in Data Science Interviews part 1
• Mastering Hypothesis T...
✔️ Cracking Hypothesis Testing Problems in Data Science Interviews part 2
Practical significance and statistical significance
• A/B Testing Analysis M...
✔️ Cracking A/B Testing Problems in Data Science Interviews
• Crack A/B Testing Prob...
📚 Trustworthy online controlled experiments, a practical guide to A/B testing
www.amazon.com/Trustworthy-On...
// Comment
Got any questions? Something to add?
Write a comment below to chat.
// Let's connect on LinkedIn:
/ emmading001
====================
Contents of this video:
====================
0:00 Intro
1:52 What are A/B tests
3:39 Why A/B tests
5:05 Steps to run A/B tests
5:20 Experiment prerequisites
6:51 Experiment design
8:03 Running Experiments
8:22 Result to decision
10:48 Post-launching monitoring

[@zenofall4455]

Emma guided me through my interview prep, and I got 3 DS offers, she had suggested this book to me, this book is awesome.

[@emma_ding]

That's awesome news! Congrats on your offers!

[@juanfelipehernandezmolina5185]

do u recommend it ? does it covers all she explained ?

[@jeoffleonora4612]

Feel like I have a lot to learn about ab testing. Can’t wait to see the next few videos!

[@nicoleli8948]

谢谢Emma! 你的视频真的很有帮助～继续加油！

[@surajbhala5298]

Thanks Emma , for making these videos. Looking forward to the next part 👍

[@lydiamai6861]

This video helps me in learning statistics and probability. Thanks Emma for making this video.

[@zishenli4195]

谢谢分享！视频条理清晰！期待更多视频！

[@tejasphirke3436]

I would love to have more of this videos.. 😃

[@zhenzhang3141]

thanks for your videos! Plz keep updating these great contents and it is super helpful!

[@katekatebangbang2435]

Emma thanks for another amazing video! 希望能看到你出更多的product sense相关的视频:D

[@zheng9470]

Hey Emma, thanks for uploading this. It's very helpful. And I've a question here: what's the difference between sample size in experiments and the number of randomization units? Thanks in advance.

[@yanliang79]

Hi Emma, thank you for the helpful video! I found on the side when you were introducing Experiment Design, the bullet points number for "size of an experiment" and "how long to run an experiment" are both 2.

[@imranm93]

Hi Emma! Great video as always. You mention that in experiment design you use seasonality, day of week, primacy and novelty effect to determine how long to run the test for. But in your A/B testing interview video you only mention sample size. I must’ve missed something but are you able to explain why there’s that difference? Thank you!

[@ankitranjan30]

Thanks for uploading this video, Emma. It' very helpful. I had a doubt regarding A/B/N testing and multivariate testing. I designed a full factorial experiment where I had 6 groups (factor A had 3 levels and factor B had 2 levels) without a control group. I checked that my residuals had constant variance and my data was normally distributed and so I was able to use Tukey's multiple comparisons. The result was that the best two groups (highest mean response indicates best in my use case) were not statistically different and even though it was practically significant as per an arbitrary self established metric, I would not recommend the winning group as it was not statistically significant (I believe this is the sanity check as the investment in winning group was worth it). I wanted to know if this problem falls under A/B/N testing or multivariate testing? Thanks in advance!

[@mariademuri2855]

This is really helpful!! Not sure how easy this is to do but I was wondering if you would be able to make transcripts available for the video. (I personally have a learning disability so it really helpful for me to have a transcript to accompany a video. Thought this might be also helpful for people where english isn't their first language too)

[@jiehu1337]

nice video, your blue sweater perfectly echoed with google color schema test!

[@thampasaurusrex3716]

thank you. See you soon!

[@user-bq6ln4ne4h]

Emma, Thank you for this helpful video! Could you please help me understand the difference between "have enough randomization units" and "size of an exp".

[@hasnanhussain6512]

Thank you!

[@zeq9021]

Love these videos on AB Testing! Any chance we can get a video on methods of experimenting when AB-Testing is not possible? (i.e. Synthetic Control methods, switchback experiments, etc.)

[@emma_ding]

Hi Eze, thank you for the feedback! I'll definitely work on creating more helpful content. Stay tuned! :)

[@manjariganapathy3574]

Hi Emma. Thank you so much for your video. Can you make a video on what to include in a resume to increase the chances of resume selection for DS interview?

[@emma_ding]

Hey Manjari, I have a video on the exact topic you are asking, here you go kzbin.info/www/bejne/roS3dKCZrb1giKs!

[@MrJohn2brown]

Hi Emma, Any book you recommend to learn Root Cause Analysis or Failure Analysis ? Thank you

[@rongzhang3242]

Hi Emma, I found a lot of case of A/B testing if about the website users. What is there is a national retail chain company that has 1000 stores around the country. And they want to test if a coupon can help increase sales. In this case, what is the randomization units here? the stores or the customer? Does A/B testing still apply in this case?

[@matt95023]

Thanks for the great video! How would you design an A/B test for a social feature on a social media site (e.g. facebook stories)? The users are interconnected with each other via posts, shares, likes, etc., rather than having their own isolated experience such as on e-commerce websites. If we randomly spilt users into 2 groups for the A/B test, users in the A group are going to be interacting with users in the B group. Could that bias the results?

[@arunas1573]

Absolutely! The way interaction effect is overcome in networks is by using cluster sampling. LinkedIn uses the concept of ego networks to identify clusters and randomization is done at the cluster level. This will minimize spill over effect. Metrics will be calculated at the cluster level and averaged at treatment and control group level.

[@purushottamraj2430]

@Data Interview Pro Please can you share all slides of A/B testing using google drive or any other application etc.

[@Squiderify]

Do you plan to make the slides available? It would be extremely useful. If so, you could also put a watermark on it because I bet it would be shared very often.

[@emma_ding]

That's a good suggestion. However the "slides" are actually video editing, i.e. part of the video. I will keep your suggestion in mind. Thanks.

[@anathemaconscience5666]

HI emma. I have a question, so why don't people do sanity check before running the experiment? So they can save energy on running an unqualified ab testing.

[@emma_ding]

Hi! Good question, we should do it before and after.

[@liruisi2196]

Hi, thank you so much for the great content! But I can not receive the "product case interview cheat sheet" even I registered my name and email through the link. How can I receive it? Thank you so much!!!

[@emma_ding]

Thank you for your interest in our product case interview cheat sheet! We have resent the email. Feel free to reach out to [e-mail] if you have any other questions.

[@Seth_majestus]

An experiment. I still dont get it. How can I estimate how long to run A/B testing?

[@stabvdhg7671]

COULD I GIVE YOU A SUGGESTION? IT IS BETTER TO ARRANGE YOUR VIDEOS IN ORDER.

[@emma_ding]

Hey Wei, this playlist may be helpful for you kzbin.info/aero/PLY1Fi4XflWSvgsaD9eXng6N5kxcMtcxGK :)