Cortex - Infinitely Scalable Prometheus - Bryan Boreham, Weaveworks

Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io
Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects
Cortex - Infinitely Scalable Prometheus - Bryan Boreham, Weaveworks
Prometheus is an excellent monitoring solution, however it is designed to store all metrics on one host in one set of files, which limits scalability and availability. This talk provides an overview of Cortex, an Open Source project proposed for donation to the CNCF, which takes metrics from multiple Prometheus instances and stores them in a NoSQL database. Outline: - Microservices architecture of Cortex - NoSQL back-end (DynamoDB, Bigtable or Cassandra) - Fault-tolerance via DHT-based ingestion tier and stateless query tier - Deployment via Kubernetes - Architectural comparison to projects in the same area, such as Thanos and M3 - Lessons learned from operating “Prometheus as a service” for two years
To learn more: sched.co/GrXL

[@ZaarShed]

A question please: since you send data to 3 ingesters you are effectively storing it 3 times, although the underlying store (s3, GCS) is already considered reliable. Does it in fact use (waste) x3 space?

[@martinschneppenheim973]

The ingesters first keep the metrics in memory before (by default 12h), so that it can compress and dedupe the metrics and then write the metrics to the underlying time series database. This saves writes and improves performance. It does that with a replication factor of 3 (default) to protect against failures of single ingester or node failures.

[@BryanBoreham]

It turns out that, if no data has been lost in transit, the three copies will coincide exactly and hence can be de-duplicated in storage. I wrote about this here: www.weave.works/blog/how-i-halved-the-storage-of-cortex