A/B Testing Mistakes to Avoid in Your Data Science Interview: Tips and Tricks!

In this video, I'm going to talk about a few mistakes that people often make when interpreting A/B testing results. You may often encounter these questions during an interview for data scientist positions. Want to know what these mistakes are and how to solve these questions correctly? Stay tuned!
Choosing the Right Metrics for A/B Testing
• A/B Testing Metrics: W...
A/B Testing "Real-Life" Example: A Step by Step Walkthrough
• A/B Testing Made Easy:...
Data Scientists Must Know: A/B Testing Fundamentals
• A/B Testing Fundamenta...
🟢Get all my free data science interview resources
www.emmading.com/resources
🟡 Product Case Interview Cheatsheet www.emmading.com/product-case...
🟠 Statistics Interview Cheatsheet www.emmading.com/statistics-i...
🟣 Behavioral Interview Cheatsheet www.emmading.com/behavioral-i...
🔵 Data Science Resume Checklist www.emmading.com/data-science...
✅ We work with Experienced Data Scientists to help them land their next dream jobs. Apply now: www.emmading.com/coaching
// Comment
Got any questions? Something to add?
Write a comment below to chat.
// Let's connect on LinkedIn:
/ emmading001
====================
Contents of this video:
====================
0:00 Overview
1:26 Data Scientists' Roles
1:48 Data Peeking
3:40 Multiple Testing Problem
8:49 Lack of Statistical Power

[@janeli2487]

Hi Emma, thanks for keeping producing such high quality videos. Do you mind explain how to deal with multiple test problems in other scenarios that you mentioned, much as sliced data into multiple segments?

[@thegreatlazydazz]

sublime video, emma. I get the feeling you will explain every useful thing in the Hippo book, thus saving us all the bother of reading it.

[@FariborzGhavamian]

Your videos are gold! Thanks!!

[@Max-yv8lw]

Hi Emma! I really like your videos. They are very insightful and helped me a lot when I was preparing my DS interviews:)

[@alexandragrishechko6618]

Your videos are super helpful, thank you!

[@Gavindsa]

Emma your videos are amazing! Keep doing what you do. Looking forward to more such content

[@kylechen4774]

thx for sharing Emma, i found these very helpful and actually thats what i encountered in real life work. mkt ran some wrong a/b testing without actually understanding it then asked me to analyze the result. maybe i should share your channel to them lol

[@snowguo1786]

Thank you! this is very helpful! Im done with this video. All the content are noted!

[@saishastech8023]

Good explanation 👌 keep going 😊

[@PikaChu-th7nb]

Great video. Very insightful. I would be interested to hear your thoughts on A/B tests set up to ensure that changes do not break anything. For example if an e-commerce website begins to place ads on the website, they want to make sure that adding ads onto the website does not cause people to buy less. What would be a good way to think about tests like this?

[@arunvaibhav5059]

You're really great!!! Please upload Facebook Data Scientist interview experience. Thanks!

[@xueyingding697]

Thanks for your videos! For using tiered sig. Lvls to check multiple metrics, how can we calculate the sample size? Should we calculate based on the most important metrics or should we calculate for each metric and choose the largest needed sample size?

[@Ayy12366]

Hi emma, this is very helpful! Thank you for making those videos, just a quick follow-up questions, I think testing multiple metrics in an A/B test is common; like usually we will pick one metric as the main metric and the couple other will just serve as support metrics; so if I just make whether or launch decision based on the significance of this one key metrics, it's fine, right?

[@Ayy12366]

The reason why I ask this is sometimes we got like tradeoff type of questions: you key metrics goes as expected but one supporting is going conflicts, will you launch, then we talk about short term and long term benefits something like that

[@zhuyanshu8941]

Can you clarify: multi hypothesis problem arise by testing a segment? Control vs. only Web segment? Or multiple treatment group such as :Control vs. Web vs. IOS?

[@user-uc3cn1jj4n]

Cannot find the material on p119 in the hippo book, the whole chapter talks about ethics.

[@jasdeepsinghgrover2470]

Good explanation but I think the last error is incorrectly handled.... Imagine you run an experiment and it is significant (you haven't checked the observed power yet), if you accept it then it is wrong but if you rerun it you just nearly doubled the p value. We should be only looking at the rerun or let the experiment have significant power (probably more than our threshold)