Custom metrics for applications in Prometheus on Kubernetes

Subscribe to show your support! goo.gl/1Ty1Q2 .
Patreon 👉🏽 / marceldempers
Checkout the source code below 👇🏽 and follow along 🤓
Also if you want to support the channel further, become a member 😎
marceldempers.dev/join
Checkout "That DevOps Community" too
marceldempers.dev/community
Source Code 🧐
--------------------------------------------------------------
github.com/marcel-dempers/doc...
Checkout my Datree video
------------------------------------------
👉🏽 • Detect Kubernetes misc...
Introduction to Service Monitors
👉🏽 • Introduction to Servic...
How to monitor Kubernetes in 2022
👉🏽 • How to Monitor a Kuber...
If you are new to Kubernetes, check out my getting started playlist on Kubernetes below :)
Kubernetes Guide for Beginners:
---------------------------------------------------
• Kubernetes development...
Kubernetes Monitoring Guide:
-----------------------------------------------
• Kubernetes Monitoring ...
Kubernetes Secret Management Guide:
--------------------------------------------------------------
• Kubernetes Secret Mana...
Like and Subscribe for more :)
Follow me on socials!
marceldempers.dev
Twitter | / marceldempers
GitHub | github.com/marcel-dempers
Facebook | thatdevopsguy
LinkedIn | / marceldempers
Instagram | / thatdevopsguy
Music:
Track: JOURNAL - Dream of you | is licensed under a Creative Commons Attribution licence (creativecommons.org/licenses/...)
Listen: / dreamofyou
Track: Loxbeats - Surface | is licensed under a Creative Commons Attribution licence (creativecommons.org/licenses/...)
Listen: / surface
Track: Reckoner - lofi hip hop chill beats for study~game~sleep | is licensed under a Creative Commons Attribution licence (creativecommons.org/licenses/...)
Listen: / reckoner-lofi-hip-hop-...
Track: souKo - souKo - Parallel | is licensed under a Creative Commons Attribution licence (creativecommons.org/licenses/...)
Listen: / parallel
Timestamps
00:00 Introduction - agenda
00:46 Overview
01:14 Source Code
02:55 Datree sponsor
04:22 Three High level components
05:25 Applications - add metrics
08:39 Starting applications with compose
10:00 Monitoring components
12:08 Create Kubernetes cluster
13:08 kube prometheus deployment
15:08 Grafana Dashboards
16:53 Deploy prometheus
19:26 Deploy service monitor
21:03 Deploy our application
22:10 Bringing it together
06:46
08:23 Check the Install
08:53 View Grafana and Prometheus Dashboards
10:00 Deploy a Prometheus
12:41 Deploy a Service Monitor
15:32 Deploy an Example Application to scrape
18:06 Check Prometheus results

[@UltimateKeyboardHero]

Over the past year or so, this has to be the 3rd time you magically read my mind about what I'm stuck with and released a video about it. Cheers Marcel 🍻

[@aceisastud]

I pass your videos around all the time. Love your attitude and great way you explain the topics. Your videos over the years have helped me countless of times. Thank you!

[@dillonhansen71]

This channel is so under appreciated. I hope the KZbin algorithm blesses this channel. This is quality content that can change peoples careers...

[@ah47f]

This tutorial is GOLD! Helped me understand a lot of prometheous mysteries! Thank you

[@madrum]

Well done! I appreciate all the time you’ve put into this video.

[@suhasraj9054]

You and Venkat are best for Kubernetes tutorials !! such a quality content ..Much Thanks for the effort you put in !!

[@weiyen]

Thanks, I watched your guide to this in regards to this and I have to say that this and I have to say that it clarified a lot of the mysteries behind prometheus and metrics. This is a very very good video and it was the video I needed to watch to understand how it works

[@metaldragon24x7]

Outstanding material ! How have ppl not found this channel yet ?

[@NivaldoPaulo]

GOLD content man, thanks!

[@thanhlocnguyen4910]

Great video. I wonder is how to show all custom metrics of prometheus intances in grafana ? Thank you...

[@mrcharm767]

i always loved all ur videos and continuing thanks for such value add

[@iposipos9342]

Your videos have helped me alot and i appreciate it. Thanks:) Will you also make a video on podmonitors?

[@georgelza]

@18:20 ... you deploy a 2nd ? Prometheus instance (running the service monitors/prometheus.yaml file) ? did I understand that correct, this separate from the one deployed as part of the kube-prometheus deployment. Some how seems to remember that deployment deployed a instance

[@SPT-bk9qc]

This is very helpful. When I follow the tutorial - I face two issues and would be helpful if you can guide me. First, there are no default dashboards populated on the grafana page, zero dashboards. Second, when I select a particular metric and see the table or graph and execute it, I do not see any values in the dashboard. Can you help me please

[@habibtahardjebbar2056]

can we have a video about Exporter query MongoDB and Prometheus on Kubernetes

[@georgelza]

For Kube-prometheus, which Grafana dashboard do you recommend/prefer ?

[@ReviewSmartTech]

Awesome knowledge shared

[@JackReacher1]

I created my own custom metrics using prom-client in python which is a counter called "request_count" for my flask app. When I tried to scale with hpa using "request_count"; in the prometheus seeing the "request_count_total" I found that only one pod is serving all the requests; instead of all the pods getting created with hpa. Marcel do you know what went wrong?

[@user-wr4jh1ul8x]

hi, were you able to find the solution?

[@chasim1982]

Great Videos as always, please make video with application slack alerting & application monitoring in grafana as well, thanks

[@avanthaditya8596]

We have a legacy app and cannot instrument it with prometheus library. The app writes metrics data to a json file on disk. How can we scrape this data in prometheus?

[@yogeshmandge1927]

How to check metrics details using CLI instead of Web?

[@do3484]

how i could save the prometheus data using volumes? I have an EKS cluster

[@user-ur4lm7rg5t]

how I can change /metrics path for each pod to my spring boot path /actuator/prometheus which has basic auth with specific credentionals?

[@MarcelDempers]

you can. service monitors allow you to change the paths and supply basic auth credentials too. See the docs github.com/prometheus-operator/prometheus-operator/blob/main/Documentation/user-guides/basic-auth.md

[@user-ur4lm7rg5t]

@@MarcelDempers thanks!

[@unkwnown1947]

Very informative, got a good help in development, Thanks 👍. Can you guide how to expose custom metrics for application written in C language.

[@eduardmart1237]

How will it work if you have several instances of your application? Like several pods..

[@MarcelDempers]

Service monitors translates to multiple endpoints as IP addresses under Prometheus targets page.

[@vinvvinv7460]

Hi, Can you please make a video & explain how to write an ansible playbook step by step in a very simple way? It's a bit confusing regarding spaces & hyphens. I think it's called indentation. In Terraform as well it's used spaces and curly brackets etc,, I am not from a programming background so face difficulty in understanding variables that are used in both applications. Please, it's a request. & Please do reply. if you can explain I would be really grateful Thanks & regards

[@user-fs8gu4ny9h]

I have a problem. At 23:31, I typed python_request_operations_total and executed it. However I always got zero value of the counter. Is it correct?

[@ad_it]

Have you exposed application service and hit application in web browser? Without hitting an application counter won't increment.

[@user-fs8gu4ny9h]

@@ad_it I forgot it but I can do custom metrics in my case now. I used the Kubernetes in docker desktop currently and realized that I need to deploy my app on Kubernetes but not in the docker. In my opinion, I may connect to the wrong app(port) and the Prometheus get the zero counter from another app.

[@180doman]

Maybe next topic: Network Policies with Calico and kube-iptables-tailer?

[@Bandude231]

Can a service monitor watch two or more services?

[@MarcelDempers]

Yes, as long as its selection \ ports match, it sure can 💪🏽

[@georgelza]

sorry to bug... it seems my prometheus-applications-0 is seeing the serviceminitor (or well i can see it under config, but it's not showing in service-discovery) and also not picking up my target. additionally... I can port forward directly to the pod, but can't get Grafana connecting to prometheus-applications-0, thinking I might need a service defined for the additional prometheus-applications-o instance ? some self diagnosing ... I've did a port forward of my container/app and I can see the metrics at ../metrics so the python and prometheus instrumentation is working, it's the service discovery where I think i broke something. it's not picking up my targets it seem. please assist. This loooooks like the absolute closest I've been to get this now working :) Appreciate this newer video.

[@MarcelDempers]

prometheus needs to be correctly configured to select the service monitor and the monitor needs to exist where prometheus is looking as mentioned in the video on service monitors. Same for the service the monitor tries to select, else you will have this issue. For grafana to access prometheus, you need a service and a datasource setup for it

[@georgelza]

@@MarcelDempers :) figure that much. but somehow with me moving app from "default" NS to "trustreg" NS I broke it... I got Grafana working. would love / appreciate if you can do example of the app not in default NS, as that will anyhow never be done in prod.

[@MarcelDempers]

It works on any namespace, just update the namespace selector. kzbin.info/www/bejne/lX_Xg554ntWYpsk

[@thanhlocnguyen4910]

I have the same problem like you. Can you look a little bit my issue ?

[@georgelza]

@@thanhlocnguyen4910 ... me still. haven't been able to get this working. haven't looked at it in a while either.

[@Babbili]

from where you got your k8s T 😆

[@BR-lx7py]

Aren't the cool kids using Opentelemetry these days?