Hidden engineering costs and complexities of A/B testing - Anastasia Petrova (NSLondon 2024.3)

In this talk, we’ll critically examine the often-overlooked engineering challenges and hidden costs of A/B testing. We’ll explore how A/B testing increases code complexity, introduces risks of data corruption, expands the testing matrix, and challenges debugging. We’ll shed light on the real impacts on developer experience and user satisfaction. I’ll provide real-life insights gained from working on products such as Depop and Monzo that might reshape your perception of A/B testing and convince you that it’s a tool that needs to be wielded responsibly and requires high engineering discipline.