A/B Testing Metrics: What You Need to Know About Success, Driver, and Guardrail Metrics!

In this video, we will take a deep dive into the fundamentals of metrics in A/B Testing. By the end of the video, you will have a solid understanding of the common types of metrics and tips for applying them in real-world A/B testing applications. Some ideas are inspired by the book Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing. I'm sure this video will be extremely helpful for both interview preparation and day-to-day work as a DS.
Blog post by Rob and me:
towardsdatascience.com/the-ul...
Part 1 of the tutorial:
• A/B Testing Fundamenta...
Part 2 of the tutorial:
• A/B Testing Made Easy:...
🟢Get all my free data science interview resources
www.emmading.com/resources
🟡 Product Case Interview Cheatsheet www.emmading.com/product-case...
🟠 Statistics Interview Cheatsheet www.emmading.com/statistics-i...
🟣 Behavioral Interview Cheatsheet www.emmading.com/behavioral-i...
🔵 Data Science Resume Checklist www.emmading.com/data-science...
✅ We work with Experienced Data Scientists to help them land their next dream jobs. Apply now: www.emmading.com/coaching
// Comment
Got any questions? Something to add?
Write a comment below to chat.
// Let's connect on LinkedIn:
/ emmading001
====================
Contents of this video:
====================:
0:00 Intro
1:13 Goal Metric
3:13 Driver Metric
4:42 Guardrail Metric
6:14 Application of Metrics
8:17 Metrics for experiments

[@emma_ding]

Typo correction: Thanks Squiderify, at 10:38 "metics" should be "metrics".

[@jackyzhang6848]

Hello Emma, thanks for your sharing, may I ask for a question, for weighted combination of, say 3 metric, how do you decide the weight among 3 metrics? For example, the engagement of FB newsfeed, the number of likes, comments and shares, there are 3 main metrics to measuring the FB newsfeed engagement, compare to like, share and comment are more meaningful to measure the interaction on FB, but how would you decide the weight among those 3 metrics in a scientific way? Really thanks for your answer.

[@SM-gf7ux]

Emma - watching your videos helped me land a data scientist position at a FAANG company. So grateful for your knowledge and your ability to share it with a wide audience :)

[@where_is_georgio]

You explain things so well, very helpful for those of us breaking into the data science world. Thank you Emma

[@tungthanhle6797]

Thanks, Emma for your contributions to our DS community! I have been enjoying your content so much so I even clicked on the like button before I watch any of your new videos :D. Thank you very much!

[@emma_ding]

Glad my videos are helpful!

[@omkarkhedkar1885]

Really intuitive videos, Thanks! Keep up with the good content!!

[@yashnikhare2210]

Amazing content and often asked in technical interview. Keep it up Emma 🙌

[@saeidsamizade6257]

This channel is gonna be a great technical resource for professionals. Thanks 👍

[@aymanz83]

Great source of information, and clear explanation for tough topics

[@tianyifang6601]

so useful and helpful!!! I learned a lot from your video! thanks so much

[@qingchen7835]

给Emma点赞！！最近面试Product Analytics 多亏看了你的视频！！讲的又好又清晰！！ 因为自己没有Product经验，所以面试还挺费劲儿哈哈~ 希望自己可以早日上岸！再次谢谢Emma的视频

[@robinhood3203]

good explanation, please keep up the good work!

[@saishastech8023]

Good explanation of metrics 😊 thanks for sharing 👌 keep going 👍

[@TrinaMoitra]

We actually use a 3rd category of guardrail metrics. Program metrics. To ensure overall quality of the experimentation program. One of the guardrails we have is ratio of iterative to exploratory tests.

[@duchen1844]

这节说的真好，微软AnE说的也不过如此。口才好，写书的Rony若是听了也要点赞。

[@doreenshen3690]

Hi Emma, thanks for the super clear and structured content! I am still a bit confused about what an "attributable" metric means. Could you add more clarity here? Thanks!

[@technicallyeasy]

i love your channel! you should create amazon referral links for the book you mentioned in your video. that way you can get referral revenue when your followers buy that book based on your recommendation. i want to read that book now

[@ranpan2861]

Hi Emma, could you dive deeper into how to answer the question “If one metric goes up, and the other goes down, what should we do?”

[@DataKnight]

Thank you for sharing

[@sparklingamber792]

Hi Emma, for the sample ratio mismatch, shouldn't we use z-test rather than t-test? I read that the variance p(1-p) and mean p are not independent for proportion comparison, so it will follow a z-distribution.

[@DataKnight]

Thanks for sharing

[@tinawang1291]

6:13 for the sample ratio mismatch check, I think we should only apply Chi-square test, which is to compare a proportion with an expected value. Can you help clarify how to use a T-test for checking SRM? Thanks

[@rubaldahat4813]

Can u shows its implementation so that it easy to remember.

[@asyvli1226]

hi Emma, thanks for your sharing, could you please share more detail about “If one metric goes up, and the other goes down, what should we do?” thanks

[@emma_ding]

Fantastic observation! When metrics go a bit wild we can experience strange outcomes for all kinds of reasons worth talking about. This has given me a great idea for more content about how to avoid analysis paralysis! Thanks for sharing your comment with me!

[@yuzhouzhang9847]

哈哈哈，你的内容真好，请问可以在Github或者medium上同步吗，给点文件啥的

[@Squiderify]

At 10:38 there is a typo: "metics" instead of "metrics". Amazing content though!

[@emma_ding]

Thanks for spotting the typo!

[@vaibhavsingh39]

Best videos

[@awang3139]

数科秘籍Emma Ding，interview rat 大救星