Bayesian A/B Testing - Marc Garcia

Description
A/B testing is a controlled experiment, where a possible improvement challenges the current version of a product. It is the most common approach to improve websites and their conversions.
Abstract
A/B testing is a controlled experiment, where a possible improvement challenges the current version of a product. It is the most common approach to improve websites and their conversions. In an A/B test, half of the users are kept in the current version as a control group, while a randomly selected half of them is presented with the challenger version. In this context, it is expected that one of the groups performs better than the other. But the important question is, is this difference in performance caused by the differences in the versions, or by randomness.
While the question may sound simple, the widely used approach of statistical significant is tricky and confusing. Some parameters need to be decided, and counter-intuitive statistics based on null hypothesis and p-values need to be performed.
But there is an alternative, Bayesian statistics. With simple techniques such as Thompson sampling, the problem can be implemented as an Artificial Intelligence system, that manages the uncertainty in the data, and adapts to it, to automatically make the optimal decision for us.
In this talk it will be covered an introduction to the main two schools of statistics, frequentist and Bayesian, and it will be shown how to implement an A/B test system based on both of them. A step by step simulation will be implemented in Python and shown, so the audience can see how both systems perform, and how they can be monitored.
www.pydata.org
PyData is an educational program of NumFOCUS, a 501(c)3 non-profit organization in the United States. PyData provides a forum for the international community of users and developers of data analysis tools to share ideas and learn from each other. The global PyData network promotes discussion of best practices, new approaches, and emerging technologies for data management, processing, analytics, and visualization. PyData communities approach data science using many languages, including (but not limited to) Python, Julia, and R.
PyData conferences aim to be accessible and community-driven, with novice to advanced level presentations. PyData tutorials and talks bring attendees the latest project features along with cutting-edge use cases. 00:00 Welcome!
00:10 Help us add time stamps or captions to this video! See the description for details.
Want to help add timestamps to our KZbin videos to help with discoverability? Find out more here: github.com/numfocus/KZbinVi...

[@morbius14]

this was a terrific presentation on frequentist a/b testing lol wish they gave him more time. Still, Ienjoyed the video Mr. Garcia

[@MaximeDendauw]

Very clear presentation. Pitty there was not enough time to elaborate on the concrete use case or reinforcement learning topic.

[@CS_n00b]

when would one use alternatives to the beta distribution?