Good Q! It depends on how the A/B test hypothesis was put together. It's a bit more complex question than you would first think. The short version: Actually when you design an A/B test, you come up with a *null-hypothesis*, that you want to *refuse*. In this case the null-hypothesis could be: 1) "B is not better than A" --» if you can refuse that, than B is better than A --» but if you can't, than B is the same or worse than A. Or null hypothesis could also be: 2) "A is better than B" --» if you can refuse that, than B is the same or better than A --» if you can't, than of course A is better than B. I think, most A/B testing tools measure the null-hypothesis version that I put in 1) (it really depends on the tool, though and I haven't used these point-and-click solutions for a while) -- the dif between 1) and 2) seems subtle, but it's real, and the statistical calculation that these software use behind the tests depend on which null-hypothesis they use. Point is: you rarely see actually neutral results. You can see better, worse OR statistical-significance-didn't-reach-treshold tests. If it's neutral, than it's potentially worse, too. BTW, you are also right by saying, it depends on the risk tolerance of the organization, too.

Actually, it is not statistical significance. 78% is the conficence level in this case. You can yse VWO calculator to calculate it. Control (Number of Visitors=16401, Number of Conversions=392), Vatiation (Number of Visitors=16165, Number of Conversions=408). With that you get p=0.218 (statistical significance). Confidence level = 1 - Statistical significance. 1-0.218=0.78 Here we have 78%

@@ekaterinavolkova4348 useful)

I think you have to know what change you have in your variant. Small changes = low MDE, because then you can detect that there is an effect. Best to use the Speero calc. When you have a high MDE, then you know that you must implement a big change in de variant to detect an effect.

Thanks for the answer, especially Speero seems very good. However, I’m having difficulty understanding how I can expect a change(a numerical number) with a business action I plan to take (a concept, an idea) if I have not enough test experience. In other words, if I want to implement a new feature that I haven’t tested before, how can I know what to expect minimum?

Hey is it good to learn a/b testing from here for product management?

@@anshikajain8389 Yes. : )

Thanks a lot, mate! Just when I thought that nobody's watching these A/B testing course videos! : ) Your feedback means a lot to me! ✌ Glad, the video helped!

Thanks!