How Thanos Almost Snapped $100,000 from Our Infra Budget - Shubham Srivastava & Ankur Rawal, Zenduty

Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon Europe in Paris from March 19-22, 2024. Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at kubecon.io
How Thanos Almost Snapped $100,000 from Our Infra Budget - Shubham Srivastava & Ankur Rawal, Zenduty
In a galaxy not so far away, where data is as vast as the cosmos, our team was troubled with observability data chaos. Seeking some clarity, we sought salvation with Thanos and Fluentbit - fabled titans against our metric storage and logging issues. Thanos empowered us with a Prometheus setup with high availability and virtually infinite historical data storage. Prometheus ascended to new heights, flawlessly scaling horizontally while Thanos Compactor's downsampling abilities promised faster results for querying older data. Fluentbit made collecting, filtering, and outputting logs across multiple sources and destinations effortless. But, little did we know that even the most powerful tools, when not wielded correctly could be double-edged Infinity Stones. Join us on a thrilling tale of blunders as we recount some missteps in configuring these tools, easily missed caveats in data downsampling and log storage, and how the pursuit of seamless data handling almost cost us over $100,000.