Easy as ABC: A Quick Introduction to Bayesian A/B Testing in Python (Will Barker)

A/B testing is a valuable and in-demand skills that data analysts, BI developers, and data scientists have in their analytical toolkits. This beginner-oriented talk will explain the basic intuitions and statistical theory behind A/B testing and showcase a simple implementation in Python.
Presentation page -- 2018.pycon.ca/talks/talk-PC-5...

[@stupedigo]

This is actually the best bayesian ab testing intro!

[@matthewturner639]

True, this is solid and simple

[@olesiaaltynbaeva4132]

100% true

[@kougamishinya6566]

The advantage of Bayesian approach vs frequentist is that you don't have to waste time and money testing more than you need to reach the most effective solution. You update the distribution as you go along and eventually the less effective options are selected less frequently leaving you with the best performing option. In frequentist you have to test each option X number of times. Even if option A is performing badly after X/2 number of trials, you can't start selecting it less like happens in Bayesian automatically. So Bayesian is more efficient.

[@Stefan111189]

Very nice and brief introduction. The relative increase calculation based on Monte Carlo is something great that I haven't seen before.

[@kinwong3224]

Beside the AB testing, Will has a very nice and simple explanation of Bayes

[@TK-pe7sf]

Good job, very followable presentation!

[@estanislaomariaferrer9]

Loved it!!! Thanks for sharing this content

[@albusdd1225]

Very informative and an amazing session

[@BillusTinnus]

Great presentation !

[@sarahwilde3501]

I found this really helpful! I haven't dove in too much to Bayesian inference yet, but this is the best intro i've seen since others are too complex to really get me started. one question I have tho....the way the 98% probability was likened to a p-value of 0.02....that's incorrect, right? they suggest similar things, but they are not the same. based on the speaker's comment at the end that he doesnt know frequentist approaches, i'm going to assume i'm right here....but i'd love other comments.

[@russellwarden2899]

You are correct. A P value is not a probability

[@arturleperoke3205]

@@russellwarden2899 hey guys.. I did not watch until the end, yet I skimmed through some comments.. I am fairly certain that we do can interpret the p-value as a probability. It is the probability to see a difference in the statistical summary between two groups as or more extreme under a specified statistical model(Wasserstein RL, Lazar NA. The ASA's Statement on p-Values: Context, Process, and Purpose. Am Stat. 2016;70(2):129-33.) Though his comparison of p-values to the 0.02 seems to be wacky.

[@leassis91]

can you explain futher why the parameters of neutral and skeptical were halved?

[@fanyin9056]

Very nice tutorial. I guess my question is (5:16) - if eventually we are gonna link the posterior probability with p-value, why do we want to conduct Bayesian A/B test at the first place?

Where’s the code ?

[@eightrice]

5:00 what does the y axis refer to? density of what?

[@sangramkapre]

posterior prob for metrics from variants A and B

[@brittosabu8401]

Please provide the source code