Don't Make These 6 Prometheus Monitoring Mistakes | Prometheus Best Practices & Pitfalls

[@rishabhverma2631]

Very happy that i came across your channel. I am quite new to Prometheus and your content has already helped me out a bunch. Thanks Julius

[@90hijacked]

I am really hyped to find your channel, been working with prometheus for a few years now, it was my primary reason to begin learning golang also instantly subscribed, I'll be lurking here on my weekends! ♥

[@BlackUnicornVlogs]

YAYYY!!! It's awesome to see how this series is progressing. Another winner 😎

[@Jam-ht2ky]

This would definitely make a great series!

[@mickiXXX]

Thanks Julius for all your videos. They are very useful for me. 👍

[@R0SS0]

Thanks for the video! Very informative

[@rafiraf1522]

Great content, thank you for recording it!

[@GeorgiKobilarov]

Great video! Thanks Julius

[@nikitaruy8619]

Thank you for the video! :) Very good content!

[@YuruCampSupermacy]

Loved the video. Looking forward to more.

[@theophani]

Ahhh!! All mistakes I’ve encountered! (including ones I made …). Great summary. I’m sure I’ll refer back to this one.

[@PromLabs]

Ohhh that's good to hear that it's relevant to pitfalls you encountered as well, thank you! :)

[@vnavalianyi]

Thank you for great videos!

[@fauxz3782]

Your videos are so good

[@jhonsen9842]

Great stuff

[@aminjafer4535]

Awesome channel

[@bulmust]

Nice video and channel.

[@prinzgonzo6646]

Hey Julius, thank you for your video. Which metric type would you propose to avoid cardinality bombs in case I want to record labels that can have lots of different values? Is there a way to change the grouping mechanism in a way that prometheus will not create a new time series for every label combination?

[@PromLabs]

A time series is identified by its metric name and unique set of labels, so any addition, removal, or change of a label will automatically mean that it's a different / new time series. So there's no way to not create a new series for a new label combination. Other than that, the only thing that matters is the total number of series generated by a metric, which results from the combinations of label values it can have. The metric type doesn't have any impact on that (and mostly just depends on what you are trying to measure), except that of course e.g. in the case of histograms, there's always an automatic "le" bucket label that gets multiplied into the total cardinality, since every custom label combination also has one series per configured bucket (so one strategy is just to configure fewer buckets if histogram cost is becoming an issue).

[@PromLabs]

Let me know what other kinds of Prometheus pitfalls and best practices you'd like to learn about! These were some technical ones that I've mentioned a lot in talks, but there are surely many others.

[@suparna20100]

Hi, just now started using prometheus. I am trying to set rate function for counter.. so i have a counter metric and the values is 1.. when i use rate for counter rate of first entry a at that time starts with 0..why for the rate of first new single instrumwntation entry starts with 0. How to resolve that..i want rate /changes/ increases should starts with non zero and then it should become 0 when its idle how to achieve this

[@stephennfernandes]

hey julien, i am pretty new to prometheus. i want to know that can we also track user level behaviour in prometheus or would i need to use a total separate tooling for that. i wanted to know what my users are doing, and watch indidividual user behaviours

[@PromLabs]

If you have more than a handful of users, then a metrics-based monitoring system like Prometheus is probably not the right choice. Imagine you have a million users and you want to track one metric about each. Then you already have 1 million time series just for that. The total budget for a big server is usually a couple of million series, so this will only work if you have very few users or you don't track a lot of info about each one :)

[@XavierTibudon]

Awesome content, Julius. I find it extremely useful, but it took me 2 months to finde it! I'll do what I can to spread the word through my channels. Insanely popular as it is, most implementations I come across during SRE consultancy engagements, only exploit a small fraction of its value. All the best, @XavierThibaudon