GrafanaCONline: Prometheus rate queries in Grafana

This was a session GrafanaCONline 2020. All on-demand videos can be found here: The GrafanaCONline schedule is here: grafana.com/ab...
Which range to use in a Prometheus rate query is already a bit of rocket science. When Grafana enters the game to visualize the result of such a query, things get even more interesting. In most cases, you’ll end up with a dynamic range, but navigating the many available settings for it is sometimes a bit daunting. But do not despair, this talk will teach you all you have to know, even in case when you add to the mix recording rules, which are blissfully ignorant of any dynamic range requested by Grafana. If all goes well, we can even inaugurate a nifty little new feature in the Prometheus data source that will improve your rate visualizations even more.

[@belmarkcaday7916]

Great explanation. Thanks ser Jaime Lannister

[@romanmelnyk7740]

This is the best explanation for rate func I watched so far, thank you so much for this video!!!

[@romankhavronenko6569]

I like this new white-board Grafana interface!

[@briggsezra7783]

Instablaster.

[@IvanyaKosmos]

used prometheus for years and only now i was blessed by algorithm and found this video. great tutorial, cleared many things for me

[@rrr00bb1]

Signal processing: Nyquist's Theorem. You must sample at a rate of 2x the highest frequency to detect. When you average over an interval, you get a low-pass filter (that filters out high frequencies. When you expect to miss 1 scrape, it makes sense that you want to use 4x because of this.

[@pythonsriracha4944]

Great explanation, thank you!

[@ruslanvs]

This is gold.

[@johnjames_cowperthwaite]

Many thanks Bjorn, great explanation!

[@prabhakarp9861]

Great talk!

[@TheASHOKNAN]

fantastic explanation

[@ankitbhalla4813]

great tutorial !!

[@ridakejji6837]

Great thank you !!!

[@MsKinder123]

Really useful:) thanks

[@micdestefano]

Thank you very much for this great explanation. I have a question if you don't mind. Let's assume I have a Prometheus query like the one you showed, e.g. rate(process_cpu_seconds_total{...}[5m]). Let's assume that now I would like to extract the 95th percentile hour by hour of the resulting time series. Is it possible to do that? I mean, how can I combine the "quantile_over_time" with rate(process_cpu_seconds_total{}[5m]) ? I've not managed to do that up to now. If it is possible and you know how to do that, can you tell me, please?

[@haralc]

Is it just me or prometheus is a bit strange? I wrote an application for testing that increments a counter by 1 every 5s. That said, prometheus can only show me .2 when apply the rate, even if I set the "step" as 5 (seconds)... this is quite annoying because there's no 0.2 request per second ... like you cannot split a human into smaller pieces.

[@IvannSerbia]

Thats how it works. The idea is to detect spikes, and it will work!