SREcon24 Europe/Middle East/Africa - Dude, You Forgot the Feedback: How Your Open Loop Control...

Dude, You Forgot the Feedback: How Your Open Loop Control Planes Are Causing Outages
Laura de Vesine, Datadog, Inc.
It's a strong principle of good UX design that users should get feedback about the results of their actions, to help prevent errors. Experienced SREs know to build in additional observability to systems to watch our systems change as we mutate them, but these are typically out-of-band and require a conscious, deliberate action to observe -- so getting good feedback into our actions requires constant vigilance and training of new users. What if we instead built control planes that tell us exactly what we've done, and what effect that is having?
This talk explores various patterns of "fire and forget" control planes in production systems, how each one contributes to outages, and some simple solutions to build better tools for operations.
View the full SREcon24 Europe/Middle East/Africa program at www.usenix.org...