Tales from an eBPF Program’s Murder Mystery - Hemanth Malla & Guillaume Fournier, Datadog

Tales from an eBPF Program’s Murder Mystery - Hemanth Malla & Guillaume Fournier, Datadog
On a beautiful fall evening, when Cilium was busy processing packets, some of its eBPF programs were murdered, leading to connectivity issues across the cluster. Join detectives Hemanth & Guillaume as they narrate tales from a multi-week investigation involving Cilium and eBPF. As eBPF continues to grow in popularity, this talk serves as a cautionary tale with pitfalls to avoid for developers who intend to build systems that need to co-exist with other eBPF-based components. This talk will include a quick overview of the Linux traffic control subsystem and how you can program it with eBPF. You’ll then get a walkthrough of how the speakers identified the root cause of the incident down to a race condition and the function guilty of the murder. You’ll learn to take advantage of bpftrace and its tools for your debugging needs. Lastly, the talk will conclude by discussing some of the ongoing work of eBPF program ownership, how it impacts security use cases and why solving for it is crucial.