A/B Testing in Data Science Interviews by a Google Data Scientist | DataInterview

👉 Looking for a comprehensive AB testing course? Visit www.datainterview.com/
👉 Join the Data Scientist Interview Bootcamp: www.datainterview.com/bootcam...
====== ✅ Details ======
🤔 Ever wondered how to ace the A/B testing questions in data science interviews?
Explained by Dan, a former data scientist at Google, explains how to design an A/B test based on a real-life example. The procedure and tips covered in the video are insightful for data science interviews at top companies such as Google and Meta.
Dan, the host, was a data scientist formerly at Google and PayPal. He launched datainterview.com/ to help candidates like you eliminate frustrations about the data science interview process and increase your success.
As an interview coach, Dan helped several clients land their dream jobs as IC and managerial DS roles at top companies such as Google, Meta, Amazon and such. Message him at [e-mail] for help!
👍 Make sure to hit the like, and check out datainterview.com/
====== ⏱️ Timestamps ======
0:00 Intro
1:02 7 Steps in AB Testing
3:02 Step 1 - Problem Statement
4:44 User Funnel
6:09 Success Metric
8:06 Step 2 - Hypothesis Testing
9:41 Step 3 - Design the Experiment
11:34 Step 4 - Run the Experiment
12:32 Step 5 - Validity Check
15:13 Step 6 - Interpret Results
16:13 Step 7 - Launch Decision
====== 📚 Other Useful Contents ======
1. Principles and Frameworks of Product Metrics | KZbin Case Study
Link: / principles-and-framewo...
2. How to Crack the Data Scientist Case Interview
Link: / crack-the-data-scienti...
3. How to Crack the Amazon Data Scientist Interview
Link: / crack-the-amazon-data-...
====== Connect ======
📗 LinkedIn - / danleedata
📘 Medium - / datainterview

[@SM-gf7ux]

Dan - watching your videos helped me land a data scientist position at a FAANG company. So grateful for your knowledge and your ability to share it with a wide audience :)

[@raamstarraamstar5115]

Can u please share me. What’s way you followed to land a job in FAANG please

[@grabngoinfo]

This is one of the best videos about AB testing interview prep on KZbin. Great job and thanks for sharing!

[@adazhu8211]

This is the best video I’ve seen about A/B testing. This is exactly a day to day DS work in tech companies. I very appreciate your sharing and looking forward for your new videos. Wishes you all the best!

[@phamvinguyen]

This is the best video about A/B testing in practice I've ever seen. Thank you for sharing

[@masonlee8573]

This is a great video step-by-step explanation and logics through the A/B testing setup! I am a recent DS bootcamp graduate and was struggling with this topic and this video helped tons! Thanks Dan

[@alicel6813]

Love the content! So much valuable insight into A/B Testing! Keep sharing more content.

[@blueberryblue2520]

Your English pronunciation is the most comfortable and tolerable throughout KZbin. Thank you so much for that. You saved my life.

[@skid-ed2qk]

Thanks for sharing! This video on A/B testing is hands down the most informative and practical one I've seen on KZbin.

[@user-uc2tq4ng6z]

The best video about A/B testing, definitely!

[@princeok12]

Extremely helpful. Thank you for this step by guide to A/B Testing.

[@iLoveTurtlesHaha]

Bro I have an interview in the morning and your video is giving me the confidence I need to go through with it. 😅

[@ustckewen]

Thank you very much, Dan! Very Helpful.

[@lightmusicable]

Thank you for the great content, really loved it. It will be great if you could have some videos around most commonly used statistical tests in depth and A/B testing pitfalls. Will keep watching！

[@martingarcia3999]

your channel is GOLD! THANK YOU

[@vaibhavgedigeri7289]

This video literally made me subscribe to you channel Dan! I have gone through countless materials and research papers on experimentation and by this is one of the best summary of the steps! Great work and I am excited to see more videos from you(especially around pitfalls for multiple metrics and more)!

[@DataInterview]

Thanks Viabhav! I created the lesson based on what I wish I had when I first learned about AB testing a couple years ago -- Dan

[@TuanSihan]

I'm currently preparing for Data Scientist interviews and this video dropped right on time. Thank you so much for such amazing content, Dan!!

[@cococnk388]

How was the interview?

[@TheYashRathi]

This is such a great video. This video is enough when you are preparing for analytics experimentation interviews

[@shounokbanerjee4757]

please make a video on how the hypothesis is being tested and we get the p value/CI

[@hemalitanna]

I am a product designer, and this is so great to learn. Thanks for explaining it in detail in a simple to understand way!

[@DataInterview]

Thank you!

[@ryanmitchell2208]

This video made me subscribe to your channel. Extraordinarily detailed explanation of all of the things that go into experiment design.

[@rishabagrawal5332]

Great content. Simple concise explanation.

[@TadBitGains]

I was asked a similar A/B testing question on a Data Science interview with LinkedIn. Had I seen this video a week prior, I would’ve made it to the 3rd round. Better late than never. Great content! I just subscribed.

[@DataInterview]

Best of luck with your prep! -- Dan

[@rezarafieirad]

great work. i love this course and it helped me alot.

[@madluvx]

Thank You for the concise explanation.

[@DataInterview]

Of course! -- Dan

[@surekhakatreddy6400]

A very detailed explanation to get started on A/B Testing. Thank you so much.

[@DataInterview]

Of course!

[@surekhakatreddy3531]

@@DataInterview Can you please explain difference between Frequentist and Bayesian approaches in A/B Testing.

[@apoorvasrini2196]

This video is awesome and really helpful if anyone wants to learn AB testing but doesn't have the time to finish the entire popular Udacity course. Thank you for making it so clear and interpretable!

[@DataInterview]

And that’s why I made this video. Ran into a lot of frustrations when I was learning AB testing myself.

[@kesemadi8075]

Great content! Thank you!

[@navkaransinghbrar4380]

🎯 Key Takeaways for quick navigation: 00:00 *🎯 AB testing is vital for data science interviews at top companies.* 01:09 *🛠️ Seven steps of AB testing: Understand problem, define hypotheses, design experiment.* 03:00 *🛍️ Example: Designing experiment for an online store, clarify success metrics.* 05:31 *📊 Success metrics must be measurable, sensitive, and timely.* 08:00 *🧪 Establish hypotheses, set significance level, plan experiment.* 09:38 *📐 Design experiment: Determine sample size, run for 1-2 weeks.* 11:30 *🕵️ Check validity: Perform sanity checks, ensure randomization balance.* 14:44 *📈 Interpret results: Analyze metric direction, P-value, and confidence interval.* 17:23 *💡 Make decisions: Evaluate scenarios based on lift and confidence intervals.* Made with HARPA AI

[@JayAgaskar]

Amazing explaination and super concise, love it

[@DataInterview]

Thanks man! - Dan

[@Y14km]

hi dan, first of all thanks for all the great work. Couple of questions on my end: -In another video (spotify case) you mentioned success metric shouldnt be top line such as retention. However, most PMs would like to see improvement in KPIs like retention. If success metric isnt something like retention, how do we guarantee the link between success metric and topline KPI (such as does improvement in CTR also improve retention in long/short term? -another question is say we saw a signicant change in country a, but no change in country b. Since unified product would be more desirable(it doesnt make sense to roll out different versions of product to different countries) how do we make the final decision here?

[@user-hw8gx9vh5v]

Hi Dan! Thanks for this great video. In the end, when you compare practical significance, do you compare the experiment Lift value with the confidence interval? Is this possible even that the formula of confidence interval uses absolute difference?

[@aravindkramesh]

*Please upload more A/B testing videos. This topic is too confusing. Could you please upload a video explaining the power of hypothesis test and how to choose the sample size for an A/B test?*

[@mingzhouzhu4668]

this is so helpful

[@yifengwu5893]

Great video. But I have a question. Isn't Step 5 Validity Check ought to be done before running the real experiment?

[@ayushsengar4153]

Which statistical test did you use to get the p-value while interpreting the results

[@sunstar123]

Same question. How did he get p value as 0.001

[@mengfanying4781]

Hi Dan, for step 6, how do you get the confidence interval (3.4%, 5.4%)? Thank you.

[@friendsplain]

@datainterview This is an amazing review of A/B testing in interviews! Quick question, if we have a minimal sample size value calculated, why do we also determine a duration of an experiment? I.e. if sample size is calculated to 500 users with searches per group, shouldn't the test duration be how long it takes a business to reach 500 users during the test period? And not an arbitrarily chosen time such as 1 to 2 weeks?

[@samw2066]

Excellent video. Should the candidate list the guardrail metrics way before step 7 (i.e., step 1) so those are known & agreed upon before the test starts ?

[@TheElementFive]

Excellent content as always Dan! As an expansion on the topic, could you consider doing a video on variation reduction techinques (e.g., CUPED, MLRATE) ?

[@DataInterview]

Yes!

[@ayushbhandari5771]

Hey Dan, really great video. One question. What will be the possible metric if we want to test a recommendation system for a fee i.e. for items shown on homepage rather than when search happens.

[@alexandragrishechko6618]

Clear and helpful, thank you

[@DataInterview]

Thank you!

[@shivanshugupta6503]

Can you explain same concept using jupyter notebook where you are following all the concepts that you talk in this video.?

[@BryceChudomelka]

This is amazing. Thank you. I will send you a Starbucks gift card if I get the job.

[@kashi1504]

This was really good

[@yoggita]

Respect !! ❤️

[@andresfelipehiguera785]

Excellent video!

[@Whateverrrrr-no5em]

the best a b testing video

[@jashanmalhi9791]

Hey Dan I am currently in my final semester of Data Science undergrad degree and am graduating a year early. I unfornately do not have any internships, but do have some projects. I am a little confused on what do after I graduate. Whether I should start applying to jobs or do a bootcamp. Please any advice will help.

[@alecryan8220]

Hey Dan, can you talk about the decision to pick revenue-per-user-per-day? Doesn't using this metric require the delta method or bootstrapping if randomizing by user? Thanks!

[@itsleomarx]

Do you think that I need to get a course on data science first and then your course or I can jump straight into yours?

[@nothingspecific314]

Hi Dan, Thanks for the video. 1. Isn't it inappropriate to compare the orders per user per day of treatment vs control? What if a big proportion of the treatment group do not use the treatment (don't use the searchbox driven by the new search algorithm)? 2. How do you define orders/user/day? is it = (total orders)/(total users x days run) or do you define the statistic at a user level and generate the distribution of means? 3. How would you assess the influence of position and popularity biases on your results?

[@vibzee4118]

Nice questions, now I am also curious to know the answers On proportion of groups not using search box, my thought is, we may take care of that by checking the stage at which user is entering in funnel

[@kangel0909]

Hmmm, revenue per day per user. That seems much more variable than conversions per user sessions, which is binomial with bounded variance. Especially for clothing on an ecommerce site that isn't Amazon. Can you help me understand why you chose this as the primary success metric?

[@aakashdusane]

oh! I was thinking the same thing!

[@adithyajob8728]

awesome

[@hao-chuanhsieh8586]

Wouldn‘t it violate the independent assumption if we use revenue per user per day for a/b testing? Since a high portion of low engaged users would just keep not buying anything.

[@aakashdusane]

What if we have same user making multiple purchases across the day? Does it make sense to have metric as revenue-per-session / revenue-per-search if we have multiple purchases by same user in a day, and metric is revenue per user per day, wont we have to change the randomization unit?

[@saeedesmailii]

Great video and summary. Just note that you've made a subtle mistake in explaining the novelty effect. The novelty effect in interpreting A/B test results refers to the impact that the introduction of a new feature or change may have on user behavior simply because it is new or different, and it's not about differentiating between new and existing users.

[@kinder458]

Excellent content, I would appreciate just a slight improvement. When you've assumed you want an alpha of 5% (in this case 2.5 per side) and you've also determined the beta. There is a different equation that gives you an output of how many people the experiment should be performed according to those assumptions. i don`t now wich one is better but from what i see is that yours doesn`t take into account the d of the experiment more importantly, you can`t always use the sigma, it depends on several cases like the size of the test, independency of the samples and his unbiased estimator the variance could be sigma OR S^2 OR sp OR SD n>=(Sigma * (Z(1-a) + Z(1-Beta) / miu0 - miu1)^2

[@anirbansen7132]

Thanks for the great video. I have a question w.r.t the sample size that you mentioned. With a 50:50 split on a website, there will be numerous sessions coming in. So, is the sample size the minimum number of sessions we need on each side to run a test. Or do we randomly sample X samples from all the incoming sessions, X being the sample size?

[@DataInterview]

Randomly sample users, not sessions.

[@randomancy]

In an A/B interview, is it common to need to calculate the Z-value or interview endpoints manually?

[@sneharaj2152]

Why is this a two sided hypothesis test instead of a straightforward one sided test?

[@minhmuic]

How do you get the p value 0.001 ?

[@beyondtheclouds95]

when you increase power (1-beta), you are actually increasing recall not precision

[@cococnk388]

Very interesting ..well understood except the launch decision part .. the diagram with horizontal lines at different levels.

[@agustinnegri6273]

Hi. Thanks for the Great video. At the moment I'm stuck with sample size, conversion rate during the test (let's say 2 weeks) and getting a histogram of the data. After 2 weeks I'll get one conversion rate. How can I get an histogram from only one data. "Inside" the CR there is a huge N (let's say 1000 users). I'm confused how can I extract the data from 1000 users to make a CR histogram. Could you help me out? Cheers

[@MashiroRedo]

Amplitude or have a data engineer/swe get it for you.

[@gemini22581]

What is lift?

[@SumedhaNagpalxyz]

Hi Dan, Very informative video! Would you have suggestions on how to select randomization units for B2B products like data bricks and slack, which have a many to 1 relationship between users and accounts?

[@DataInterview]

Randomize at the account level - sort of like cohort-based randomization. So users within the sample cohort is exposed to the same variation condition.

[@ONoesBird]

Why do we use a two sided hypothesis test here? Can we not assume the new algorithm is at least the same as the old one and use the extra power?

[@juanmora537]

You're allowing for the possibility that the new ALGO might actually hurt your KPI goals

[@thomasanderson857]

Can you explain how you got p-value = 0.001 and confidence interval = [3.4, 5.4]?

[@monikakhanchandani2742]

I have the same question

[@samiyehmahmoudian9673]

I have the same question too

[@pal999]

We're all Waiting for the reply @Dan

[@yankeephan87]

@datainterview

[@abdullahfaizal7739]

Your explanation is great but the you could avoid the background music. Infuriating!...

[@robertwilsoniii2048]

Welp, these videos tell me I am qualified to work at google in data science. Nevertheless, I don't see job openings in 2023. And I dropped out of school, so I've never been offered an interview despite my ability in statistics.

[@robertwilsoniii2048]

On the other hand, I'm not sure I really care about how effective ads are and getting users to make purchases. I'd rather study other things like whether or not certain types of jobs accelerate biological aging non-linearly.

[@abdibabu177]

I ni cash

[@davidbrown6612]

??????? ?