Custom metrics for applications in Prometheus on Kubernetes

[@UltimateKeyboardHero]

Over the past year or so, this has to be the 3rd time you magically read my mind about what I'm stuck with and released a video about it. Cheers Marcel 🍻

[@airrloww]

You're my hero Marcel, I started this job three month ago and you're the reason I'm not getting fired. I really appreciate everything you do

[@weiyen]

Thanks, I watched your guide to this in regards to this and I have to say that this and I have to say that it clarified a lot of the mysteries behind prometheus and metrics. This is a very very good video and it was the video I needed to watch to understand how it works

[@aceisastud]

I pass your videos around all the time. Love your attitude and great way you explain the topics. Your videos over the years have helped me countless of times. Thank you!

[@dillonhansen71]

This channel is so under appreciated. I hope the KZbin algorithm blesses this channel. This is quality content that can change peoples careers...

[@ah47f]

This tutorial is GOLD! Helped me understand a lot of prometheous mysteries! Thank you

[@mtofani91]

You're the best my friend, i was around this scenario last year and i didn't realize how to scrape / autodiscover the svc/pods. Thanks again!

[@suhasraj9054]

You and Venkat are best for Kubernetes tutorials !! such a quality content ..Much Thanks for the effort you put in !!

[@madrum]

Well done! I appreciate all the time you’ve put into this video.

[@metaldragon24x7]

Outstanding material ! How have ppl not found this channel yet ?

[@georgelza]

@18:20 ... you deploy a 2nd ? Prometheus instance (running the service monitors/prometheus.yaml file) ? did I understand that correct, this separate from the one deployed as part of the kube-prometheus deployment. Some how seems to remember that deployment deployed a instance

[@mrcharm767]

i always loved all ur videos and continuing thanks for such value add

[@thanhlocnguyen4910]

Great video. I wonder is how to show all custom metrics of prometheus intances in grafana ? Thank you...

[@NivaldoPaulo]

GOLD content man, thanks!

[@ВадимНич]

how I can change /metrics path for each pod to my spring boot path /actuator/prometheus which has basic auth with specific credentionals?

[@MarcelDempers]

you can. service monitors allow you to change the paths and supply basic auth credentials too. See the docs github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/basic-auth.md

[@ВадимНич]

@@MarcelDempers thanks!

[@avanthaditya8596]

We have a legacy app and cannot instrument it with prometheus library. The app writes metrics data to a json file on disk. How can we scrape this data in prometheus?

[@miguelconcepxon]

how does garbage collection work in the Prometheus metrics client? mine eventually hits oomkilled

[@SPT-bk9qc]

This is very helpful. When I follow the tutorial - I face two issues and would be helpful if you can guide me. First, there are no default dashboards populated on the grafana page, zero dashboards. Second, when I select a particular metric and see the table or graph and execute it, I do not see any values in the dashboard. Can you help me please

[@habibverse]

can we have a video about Exporter query MongoDB and Prometheus on Kubernetes

[@zuowang5185]

Do you need a service-monitor to scrape?

[@陳子鴻-p4d]

I have a problem. At 23:31, I typed python_request_operations_total and executed it. However I always got zero value of the counter. Is it correct?

[@陳子鴻-p4d]

@@ad_it I forgot it but I can do custom metrics in my case now. I used the Kubernetes in docker desktop currently and realized that I need to deploy my app on Kubernetes but not in the docker. In my opinion, I may connect to the wrong app(port) and the Prometheus get the zero counter from another app.

[@iposipos9342]

Your videos have helped me alot and i appreciate it. Thanks:) Will you also make a video on podmonitors?

[@chasim1982]

Great Videos as always, please make video with application slack alerting & application monitoring in grafana as well, thanks

[@ReviewSmartTech]

Awesome knowledge shared

[@do3484]

how i could save the prometheus data using volumes? I have an EKS cluster

[@avneeshagnihotri6664]

Please create video for Prometheus adapter configuration for custom metrics for GPU based application scaling and monitoring on grafana with using dcgm- exporter exposes the GPU metrics , i am stuck in configMap.yaml file , i am manually create all setup of k8s cluster. Please help me k get hpa showing /50% , unable to fetch custom metrics api

[@JackReacher1]

I created my own custom metrics using prom-client in python which is a counter called "request_count" for my flask app. When I tried to scale with hpa using "request_count"; in the prometheus seeing the "request_count_total" I found that only one pod is serving all the requests; instead of all the pods getting created with hpa. Marcel do you know what went wrong?

[@AshrafulHuda-n3u]

hi, were you able to find the solution?

[@georgelza]

For Kube-prometheus, which Grafana dashboard do you recommend/prefer ?

[@yogeshmandge1927]

How to check metrics details using CLI instead of Web?

[@eduardmart1237]

How will it work if you have several instances of your application? Like several pods..

[@MarcelDempers]

Service monitors translates to multiple endpoints as IP addresses under Prometheus targets page.

[@vinvvinv7460]

Hi, Can you please make a video & explain how to write an ansible playbook step by step in a very simple way? It's a bit confusing regarding spaces & hyphens. I think it's called indentation. In Terraform as well it's used spaces and curly brackets etc,, I am not from a programming background so face difficulty in understanding variables that are used in both applications. Please, it's a request. & Please do reply. if you can explain I would be really grateful Thanks & regards

[@georgelza]

sorry to bug... it seems my prometheus-applications-0 is seeing the serviceminitor (or well i can see it under config, but it's not showing in service-discovery) and also not picking up my target. additionally... I can port forward directly to the pod, but can't get Grafana connecting to prometheus-applications-0, thinking I might need a service defined for the additional prometheus-applications-o instance ? some self diagnosing ... I've did a port forward of my container/app and I can see the metrics at ../metrics so the python and prometheus instrumentation is working, it's the service discovery where I think i broke something. it's not picking up my targets it seem. please assist. This loooooks like the absolute closest I've been to get this now working :) Appreciate this newer video.

[@MarcelDempers]

prometheus needs to be correctly configured to select the service monitor and the monitor needs to exist where prometheus is looking as mentioned in the video on service monitors. Same for the service the monitor tries to select, else you will have this issue. For grafana to access prometheus, you need a service and a datasource setup for it

[@georgelza]

@@MarcelDempers :) figure that much. but somehow with me moving app from "default" NS to "trustreg" NS I broke it... I got Grafana working. would love / appreciate if you can do example of the app not in default NS, as that will anyhow never be done in prod.

[@MarcelDempers]

It works on any namespace, just update the namespace selector. kzbin.info/www/bejne/lX_Xg554ntWYpsk

[@thanhlocnguyen4910]

I have the same problem like you. Can you look a little bit my issue ?

[@georgelza]

@@thanhlocnguyen4910 ... me still. haven't been able to get this working. haven't looked at it in a while either.

[@unkwnown1947]

Very informative, got a good help in development, Thanks 👍. Can you guide how to expose custom metrics for application written in C language.

[@Bandude231]

Can a service monitor watch two or more services?

[@MarcelDempers]

Yes, as long as its selection \ ports match, it sure can 💪🏽

[@180doman]

Maybe next topic: Network Policies with Calico and kube-iptables-tailer?

[@Babbili]

from where you got your k8s T 😆

[@BR-lx7py]

Aren't the cool kids using Opentelemetry these days?