How Prometheus Halved Its Memory Usage - Bryan Boreham, Grafana Labs

How Prometheus Halved Its Memory Usage - Bryan Boreham, Grafana Labs
Prometheus is the most popular CNCF project for monitoring, but if you have millions of metrics it needs gigabytes of memory.
And, Prometheus runs as a single process: it’s sometimes a challenge to find a machine large enough to run it.
The good news is that, over the past year, work has been done to reduce Prometheus’ memory usage, in some cases by 50%.
In this talk we’ll look at:
* What kinds of data Prometheus is holding in memory.
* How profiling lets you analyze where the biggest savings could be found.
* How transaction isolation was causing a big blow-up for the largest Prometheus.
* Why a whole new data structure was needed for the labels that identify time series.
* The impacts of these changes on related projects such as Cortex and Thanos.

[@Worldwidegam3r]

Great to see the progress from Bryan's PromCon talk in 2022.

[@pawandeepb5967]

this is awesome !!! thank you for a great talk.

[@lichir]

This is a masterclass. Thank you very much Sir!

[@AbhishekKothari638]

Fan ! This was just the most awesome storytelling

[@BryanBoreham]

😊

[@everythingtechrelated]

How can I reduce memory usage when I have ~3000 pods running with istio proxy sidecar. I have also disabled few non required metrics as well, still usage is 40GB of prometheus. We are using prometheus operator with Thanos. I tried the new version and GOGC 60% but not much helpful. Any help here is appreciated.

[@scrabblesolver8168]

I don't understand how many of these symbol tables are generated to not cause a bottleneck and according to that? This is one of the most important thing for this design choice but it was not mentioned.

[@BryanBoreham]

If you can read Go, check out the code in the Prometheus repo, PR number 12304. The main idea is there are multiple symbol-tables: one for each scraping target, one for reading each block from disk, etc. Also the table is locked for writing (i.e. when new series are added) but lock-free for reading because all the data used by an existing series is immutable.