A/B Testing & Statistical Significance - 4 Steps to Know How to Call a Winning Test

[@Arcangel3]

I checked tenths videos of A/B testing, this one deserves more likes

[@classyman5627]

Fabulous explanation and very crisp video and demonstration with example.....thank you

[@MinusPlusPlusergibtMinus]

good Job! Your focus on Data Literacy and Quality of Data is a huge plus of your Video! Thank you!

[@DavidRodriguez7]

This is incredible! Really clear, concise and easy to understand. Thank you so much!

[@TestingTheory]

Glad it was helpful

[@daniellekavanaugh7732]

This is really helpful. I learned a lot. Learning more about A/B testing for UX and the natural variance is a new and very interesting concept to me. Thank you!

[@TestingTheory]

Glad it was helpful!

[@saghababa]

Loved the video. Very useful.

[@TestingTheory]

Glad it was helpful!

[@Big-Al-Bro]

Nice vid on A/B testing and interpreting results through data quality, managerial significance, and statistical significance. Clear and concise!

[@TestingTheory]

Thanks for the comment Aleksandr

[@emmydistortion3997]

Wooow. Thank you! Really understandable. Very important topic!

[@TestingTheory]

You're very welcome!

[@molacool]

Binge-watching these AB testing videos :) ;)

[@TestingTheory]

Have fun!

[@fancyview]

Really a nice video on A/B Test, thank you!

[@TestingTheory]

Glad you liked it!

[@bipashasengupta9417]

Very well explained!

[@TestingTheory]

Glad you liked it

[@nigerianprince5389]

Great video. Really helpful mate thanks

[@TestingTheory]

Glad you enjoyed it

[@maxsteel4590]

Quite useful concepts..not everything is covered by a single number. Noted

[@TestingTheory]

Glad it was helpful

[@peerapongtoi1040]

thank you for your vdo. it's very useful and easy to understand :)

[@TestingTheory]

Glad it was helpful!

[@jennylockwood1020]

Very comprehensive videos #gratitude

[@TestingTheory]

Glad it was helpful!

[@HafizShehbazAli]

Well explained.

[@TestingTheory]

Thanks Hafiz.

[@KelseyCodes]

Could you do a video explaining lift? I’ve only seen it used in the context of data mining and association rules, but it seems like you’re using it different here and I’m not sure what this measure represents here.

[@TestingTheory]

Hi Kelsey I think this video will help you out. At 3:26 I start talking about lift. kzbin.info/www/bejne/mHjHhaVtrq-VhNU

[@KelseyCodes]

@@TestingTheory thanks so much, that helped a lot.

[@khanjandesai2467]

Loved the video

[@TestingTheory]

Glad you like it!

[@oliverpaton556]

Variance doesn't need to be consistent. If you have a fairly optimal site already results will tend to be close and you just need more volume.

[@TestingTheory]

If larger changes are made, results are less likely to be close. From my experience people running tests will make little tweaks to the site and get results that are very close. The bigger the change the bigger the difference in the result for most tests.

[@RohanGL]

Awesome, thank you!

[@TestingTheory]

Glad you like, thanks Rohan.

[@neversaygoodbyeMBV]

Great points, however, doesn't this method suffer from Peeking issues?

[@TestingTheory]

If you are peeking at your data for the right reasons, then you aren't suffering from peeking, but you are benefiting from peeking. Check out this video which explains that more. kzbin.info/www/bejne/mYSTaGWnfr6beMk

[@carlosarturogalvangrajales5365]

Thx a lot!

[@TestingTheory]

You're welcome!

[@annaroser-dummig8390]

great video! :)

[@TestingTheory]

Thanks Anna.

[@spazgrafficz6113]

Can I run this test over one full day? Great advice by the way.

[@TestingTheory]

Are you talking about running the natural variance test? You would want to get more data than just a day, but one day is better than nothing. I guess it depends on your risk tolerance for having an unknown higher or lower variance amount. Me personally, I prefer more data so I can be more confident in the numbers I am using.

[@thumbliner]

Nice video. Nicer shirt.

[@TestingTheory]

Thanks!

[@amirfakhim7797]

This is a great video, thank you. Do PMs design, implement and run the A/B testing or get help from data scientists to do it and report the result?

[@TestingTheory]

It really depends on your organization. Some organizations have have a cross functional team of experts to support this work. In most smaller organizations a person might wear multiple hats and do many functions.

[@ZacharyDudgeon-x7k]

Steadily 😎

[@oliverpaton556]

100.... wow that's low

[@TestingTheory]

Yes that would be the extreme example and if it was that low you would need to see the trend and lift doing well too. That one data point in isolation isn't enough.