Handling Billions of Metrics with Prometheus and Thanos - Ravi Hari & Amit Auddy, Intuit

Don't miss out! Join us at our next Flagship Conference: KubeCon + CloudNativeCon Europe in Paris from March 19-22, 2024. Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at kubecon.io
Handling Billions of Metrics with Prometheus and Thanos - Ravi Hari & Amit Auddy, Intuit
At Intuit we have ~320 Kubernetes clusters running with ~5000 services and ~40 addons in a cluster which generate ~2 billion active time series metrics. In this talk we are going to walkthrough how we leveraged the Prometheus and Thanos to solve the challenges we faced with these metrics at scale. We also present a demo and discuss the design strategies used to solve various use cases with metrics: 1. Live metrics at scale with Argocd Metrics extension. 2. How we have leveraged Object store to serve metrics for AIOPS. 3. How do we produce metrics into Kafka eventbus to generate golden signals with Prometheus Kafka adapter. 4. How argo-rollouts consumes prometheus metrics for Blue-Green / Canary deployments etc.,