How Not to Scale Your Prometheus in Production? - Kush Trivedi & Nikola Collins, DevRev

Don’t miss out! Join us at our upcoming event: KubeCon + CloudNativeCon Europe 2023 in Amsterdam, The Netherlands from April 17-21. Learn more at kubecon.io​. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects.
How Not to Scale Your Prometheus in Production? - Kush Trivedi & Nikola Collins, DevRev
DevRev, The World’s First Developer CRM, has seen a 10X increase in the number of microservices during the past year. DevRev, since its inception, has adapted the cloud-native way of running its microservices; with recent migration to the service mesh architecture, DevRev has a strong appreciation for the cloud-native community. After migrating to service mesh and onboarding the customers to the public beta offering, we had seen a massive increase in the metrics ingestion, our observability costs, and new dashboards on top of the custom metrics. It was a matter of a few weeks when we realized that we needed more flexibility, customization, and control of our observability setup. This talk will guide you through the journey of our learnings while adapting and scaling the Prometheus, our learnings on various sort of Prometheus management strategies we went through, and how we efficiently controlled our costs without compromising the flexibility service-owners needed to manage and visualize their custom metrics.