Beautiful Dashboards with Grafana and Prometheus - Monitoring Kubernetes Tutorial

[@TechnoTim]

Have you set up monitoring yet?

[@leonardotoschi585]

I use remote proxmox for monitoring

[@ytuser13082011]

Yes, right now thanks to your help! maybe you can also show how to add physical machines to this Prometheus Stack, so they can also get monitored?

[@JerryGoNuts]

Fantastic. I have a question. Have you ever created a K3s pod running Docker, and then run Docker inside that Docker pod??? I'm trying to figure that out.

[@andrusaren8348]

I have Rancher installed and monitoring enabled.

[@MadChristianX]

I really love your videos and appreciate your work. Using your Grafana-Loki setup to monitor my main traefik reverse proxy secured with crowdsec. Just set up a matrix synapse homeserver for my friends just behind it.

[@epochphilosophy]

Actually mind blowing how easy you make these things, whereas I was to do this on my own it would take me weeks! Thanks so much for your work.

[@AminM459]

much appreciated, thank you notes: @15:22 graphana passwords, you can do them in terminal (but they are stored in history) then you wrote to file echo > admin-user which is stored in history

[@LampJustin]

25:00 Correction: It is a statefulset ;) Statefulsets always get created with a normal name and a number at the end. Replicasets in contrast get created by deployments and always have some random stuff at the end. Otherwise great video!

[@TechnoTim]

Thank you! I added a caption but it was small and hard to see!! Good call!

[@DamjanKumin]

Tim, another “old school” video that is just fantastic! Keep it up :) ⭐️

[@roberto_camp]

Great video. You might want to consider advertising in the title that you are specifically working with the kube-prometheus-stack. This is a very powerful approach and I think it would great if your video title had some reference to that.

[@liamhardman875]

Brilliant timing, just set up my own GKE Autopilot cluster a couple days back and wanted to implement and document monitoring!

[@NarendraChennamsetty]

Incredible tutorial..this is exactly what I was looking for. Subscribed!

[@marwanfikrat7716]

This is really good. To the people who made Prometheus, to the implementers of helm and then the charts and finally you I can very easily get up and running. Thank you! Subscribed :))

[@filiecs3]

Your talk of deleting secrets from the Yaml file reminded me that I recently solved this problem in my Kubernetes cluster. Using Hashicorp-Vault paired with another project called External Secrets, I am able to automatically pull my secrets from Vault and inject them into my applications automatically. You should do a video on that so that people don't risk keeping all their secrets in source control or on the host.

[@patrickmartins1925]

This channel is fantastic! Thanks Tim for the great content.

[@TechnoTim]

Thank you!

[@ytuser13082011]

People who wrote Prometheus are sadists, just like those who wrote NixOS. Documentation is convoluted and filled with overcomplicated terms and more often than not it does not make any sense. I'm in IT field for well over 20 years now and I rarely see such a disaster. Thanks Tim for helping us to get our heads around such unapproachable and badly documented piece of software.

[@notmything6629]

I completely agree. But it happens a lot with already well known open source projects that are also used by enterprises

[@szymonsko6302]

Totaly agree :D I just handle by my own pain to setup demo with HA prometheus with thanos with 3x receivers "push" model :)

[@TechnoTim]

😂. Thank you!

[@cyber-paul]

[REDACTED] Great detailed walk-through Tim. Thanks so much.

[@nataliekruck810]

Hey Tim, your videos are so great! Keep up this amazing work! Ps: I would love an video on deploying gitlab. Seriously struggling 😅

[@lightyagami0ben]

Thanks for the great content Tim, your videos help me a lot at my work !!

[@omarlittle5802]

YEEEEEEES Tim! I've been waiting for you to cover this!!

[@kaitlynk2145]

Great video always love tutorials thanks for all you do

[@kriswachtell]

Hi Tim! Your videos are great! In my cluster the values.yaml file, unfortunately, results in the prometheus-prometheus-prometheus-0 pods continuing to crash (error log: "parsing YAML file /etc/prometheus/config_out/prometheus.env.yaml: empty duration string". It's very likely that because of a difference between the version of the CRDs and the operator version (e.g. operator version > CRD version). In my case it got solved by adding the following to the values.yaml file: prometheus: prometheusSpec: scrapeInterval: 30s evaluationInterval: 30s

[@Bear-form]

Very good. I had issues implementing it before

[@azr_sd]

looking exactly for this!! thanks

[@barryx-simten1119]

Well done! Thank you Tim.

[@TechnoTim]

Very welcome

[@dh2-io]

As always a great video! I had planed to setup the same thing you did in the vid, a bit over two weeks ago, but my laptop broke and now I have to get it repaired or a new one. #100daysofhomelab keep going

[@qzorn4440]

Wow. Looks like a good project for the Orange Pi5? 😎 Thank you.

[@RichDurso]

Tim, you talked about relabeling a bit in the config section but didn't see you cover what they did in the demo. I tried the labels you used as-is, and nothing changed. However, instead of "targetLabel: kubernetes_node" I tried "targetLabel: instance" and then some dashboards started changing instance IP address to node name. I'm using K3s with contrainerD (no docker).

[@minecraftbuildingtips]

Great video series! Question: what would be the proper way to add additional extra_server_args to a K3s cluster setup with your playbook?

[@avivlevi1055]

Super helpful video thank you !

[@TechnoTim]

Glad it was helpful!

[@Subbeh2]

Great stuff as always. Just checking where I can find the YAML file to edit the extra_server_args as mentioned in the video if you don't want to go the ansible route?

[@himanshuvishwakarma1521]

Thanks It Works.

[@Davidlavieri]

It's feature complete but i find it extremely bloated even when turn off a bunch flags

[@scigama71]

i really liked the intro... subbed :)

[@TechnoTim]

Thank you!

[@Veselchuna]

Does this helm chart need a metrics API server enabled to get (all) node/storage/cpu/RAM metrics ?

[@TatoloMatlali]

Where can we find the values.yaml file you created? Seems to be working just fine for my needs

[@benjamintischlak1951]

Hey Tim, nice Video! I`ve got a short question: Do you know if there is a script from Proxmox Helper Scripts for this setup? This would be very useful if there is a one command in the node shell posibility. best regards, Benjamin

[@TechnoTim]

Thank you! This Ansible playbook will set up k3s for you in a few minutes! kzbin.info/www/bejne/eZPOdoqZir9qsK8

[@benjamintischlak1951]

@@TechnoTim You rock 👌👍 thx

[@robmit4020]

Tim great video Can you share a sample yaml for the ingress on grafana, prometheus and alert manager to use and expose access Thank you

[@sonubhagat6372]

Could you please make a detailed video on small step ca on kubernetes. For self singed certificate

[@omarlittle5802]

Thanks

[@TechnoTim]

Wow! Thank you so much!!!

[@subzizo091]

hello , thanks Tim for the great content, please how to pass the extra args if the cluster is already up "how to update it in a yaml file " , thanks again

[@trevorrydalch8959]

You have several videos on setting up and using rancher, I'm curious why you've started using Lens. Do you still use Rancher? What led you to start using Lens and what do you like more about it?

[@TechnoTim]

I do still use rancher and love it! Use Lens sometimes at work and other clusters where Rancher is not managing them!

[@trevorrydalch8959]

@@TechnoTim I'll have to check out Lens! Thanks for the reply.

[@trevorrydalch8959]

I set up an ingress and then set replicas for grafana to 2 and then brought a node down (to simulate failover). It didn't handle it well. Lost some custom panels I had put together. Any chance you could do a video on HA grafana?

[@kishoresharma1130]

Hi , Thanks for the awesome video. Can you please share the dashboard Json which you showed in the video . That looks good. If you have that dashboard custom JSON was placed in your GitHub pls share and looking for kubernests pod specific and cluster specific dashboard

[@TechnoTim]

It's included when you install this helm chart!

[@jart122]

Can you please create a video on how to monitor cpu temperatures in proxmox with dual Xeon processors. Thank you.

[@cannfoddr]

I have watched your video and read the docs - I am not clear what needs to be done with the extra_server_args. I have an existing cluster built with your ansible. I COULD reset it and add the extra args to the ansible variables and start over. However I have done quite a bit to the server and would then need to recover everything all over again. You mention that these can be added to a server configuration and the services restarted but I cannot find where you cover that route?

[@bluesquadron593]

I have just re-ran the deployment over the old one. Seemed to work…

[@stefanopucci6313]

my question is : can i get this on proxmox lxc container ? some guides or video ?

[@dmcg7900]

Is there a dashboard that includes CPU temp (I’m running a raspberry pi cluster)?

[@bastooopanooo2311]

Same can done using Java Melody.

[@JakeSchweihs]

I got this setup in my Kubernetes cluster on AWS (EKS) and it seems to be working well except I do not see any data in my etcd Dashboard. I changed the 'endpoints' to be the private IPs of my nodes in my cluster. Is it possible that this is just a limitation of using the AWS hosted cluster? or is there any other config worth investigating? Thanks.

[@TechnoTim]

It’s likely. AWS may not expose those endpoints on etcd nodes.

[@JakeSchweihs]

@@TechnoTim You got a sub for giving me the quickest reply in YT tutorial history. Thanks.

[@saintnoahl]

How come I don’t see those automatically generated dashboard?

[@admun]

Nice tuning to the values.yml! I am using kube-prometheus-stack too for my new rke2 cluster. I deploy it with fluxCD

[@TechnoTim]

Nice!

[@MinhNguyen-mb9of]

any idea on how to import wmi importer metrics to prometheus targets ?

[@swaroopkumarkondreti5299]

values.yaml file missing where can i get ?

[@altverskov]

You forgot to post your values.yaml :(

[@UntouchedWagons]

Tim did you ever make the alerts video?

[@TechnoTim]

Unfortunately not yet! AlertManager is wildly complex!

[@giuseppebinetti87]

I have already a Grafana instance running, is it possible to send all the metrics to that and import the dashboards to that instance?

[@TechnoTim]

Yup, sure with a little config!

[@shouryashrivastava3948]

Can you Make a updated video on setting up k3s on home-lab.

[@TechnoTim]

Already did! kzbin.info/www/bejne/eZPOdoqZir9qsK8

[@SumitPal-bs7cu]

Hi, Can you please share the grafana dashboard id you used in this video?

[@TechnoTim]

It's in the docs link and ships with this chart!

[@wva5089]

Thanks for your videos. your password file idea is bad..your bash history AND your filesystem now have that password. one solution is to use environment variables. read -p "enter password:"; kubectl create secret generic mysecret --from-literal="admin-user=adminuser%" --from-literal="admin-password=$REPLY" or slightly more succinct: kubectl create secret generic mysecret --from-literal="admin-user=$(read -p user:;echo -n $REPLY)" --from-literal="admin-password=$(read -p "password:";echo -n $REPLY)" Thanks again for your videos.

[@TechnoTim]

Thanks for the tip! Agreed, there are many ways to do this!

[@RichDurso]

@@TechnoTim The echo commands you did will be in your history. With Bash, start any command line with a space and it will not be added to your command history file (typically enabled). You can do " echo hi" (no actual quotes) and press up arrow for last command.... its not there. ZSH needs "setopt HIST_IGNORE_SPACE" set to work.

[@r351574nc3]

Left out crds from video

[@pausanchezv]

Hey yo! Tutorial's good but where's the values.yaml? Can't find it!

[@TechnoTim]

It's on my docs site, just search my site for the title name!

[@pausanchezv]

@@TechnoTimThanks, I have one last question. I have my cluster on GCP, so the master nodes are kinda opaque to me. I cannot get the IPs the way you do and paste them in the values file. I'm just omitting the IPs part. Would that be correct? Thanks :)

[@WeAreTheTENDERS]

Song ID in the beginning? :P

[@MokshaDharma]

When joining Mastodon?

[@MrEric377]

Are there alerts for expiring certificates? Lol, personal problems.

[@TechnoTim]

Asking for a friend 😀

[@xamindar]

Wow this thing murdered my 5 pi4 cluster. Kubectl couldn't even reach it anymore. Pretty awful set of tools if they require that much resources.

[@TechnoTim]

Sorry, but it’s a pretty standard monitoring stack for k8s!

[@xamindar]

@@TechnoTim I guess just not for k3s on top of raspberry pis.

[@xamindar]

@@TechnoTim And I didn't mean to be rude or anything. You made a great tutorial and explained everything very clearly. Thank you.

[@TechnoTim]

Not at all! I didn't take it that way! Pis are great for learning clustering but stumble when it comes to common workloads!

[@szymonsko6302]

What about HA on this setup ?

[@TechnoTim]

just increase the replicas and you have HA

[@peterdedene]

What is your terminal font?

[@TechnoTim]

Default zsh terminal theme!

[@nlahmi]

Can you share your short values.yaml file? Thanks!

[@TechnoTim]

I did! It’s in the docs linked in the description!

[@stevemulcahy5014]

@@TechnoTim I don't seem to see the link for the docs in the description...

[@christianleicht7973]

Grafana is slow slow slow

[@TechnoTim]

Hmmmm.... mine isn't

[@santacloudy]

wtf is all this, jeez

[@Arcane_Dragon878]

huh?

[@rabhiff]

I think kubernetes sucks because it has built in Google metrics and it won't work in air gapped envs etc.

[@TechnoTim]

not sure what you mean? k3s doesn't have anything built in from Google, well, except for the fact that it was built by Google but all proprietary code is stripped out

[@rabhiff]

@@TechnoTim Hello Tim, I like your channel a lot. I meant Redhat based envs and native Kubernetes. You are right K3, Openshift and Tanzu works in air-gapped hardened enviroments. There are a lot pitfalls in docker and kubernetes, I personally prefer pure vms. Persistence, Timezones(logs), host swapping issues(docker), complexity(microservice envs kubernetes) etc.

[@rabhiff]

I mean UTC clock in logs is petty hard sometimes..

[@rog33ful]

Tim, helm install prometheus prometheus-community/kube-prometheus-stack # is not working..

[@bluesquadron593]

Did you manage to solve the issue, it didn’t work for me either. In fact it crashed my cluster…

[@grencez]

now i can get paged from home too 🥲

[@TechnoTim]

🤣

[@rog33ful]

@@TechnoTim helm install prometheus prometheus-community/kube-prometheus-stack # is not working..(Install failed). Can you pls verify. I am wondering if something needs to be done on microk8s. It worked before on one of the other machines a month ago.