Monitoring, Logging, And Alerting In Kubernetes

What is the best combination of tools for monitoring, logging, and alerting in Kubernetes?
#prometheus #grafana #loki #robusta
Consider joining the channel: / devopstoolkit
▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬
➡ Gist with the commands: gist.github.com/e593ff9fa1a34...
🔗 Prometheus: prometheus.io
🔗 Robusta: robusta.dev
🔗 Loki: grafana.com/oss/loki
🔗 Grafana: grafana.com/oss/grafana
🎬 Kubernetes Notifications, Troubleshooting, And Automation With Robusta: • Kubernetes Notificatio...
▬▬▬▬▬▬ 💰 Sponsoships 💰 ▬▬▬▬▬▬
If you are interested in sponsoring this channel, please use calendly.com/vfarcic/meet to book a timeslot that suits you, and we'll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below).
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬
➡ Twitter: / vfarcic
➡ LinkedIn: / viktorfarcic
▬▬▬▬▬▬ 🚀 Courses, books, and podcasts 🚀 ▬▬▬▬▬▬
📚 Books and courses: www.devopstoolkitseries.com
🎤 Podcast: www.devopsparadox.com/
💬 Live streams: / devopsparadox
▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬
00:00 Introduction to monitoring, logging, and alerting
00:35 Metrics And Alerting With Prometheus
08:04 Notifications With Robusta
10:01 Logs Collection With Loki
13:45 Dashboards With Grafana

[@DevOpsToolkit]

IMPORTANT: I made a mistake in the video by saying that AlertManager is querying Prometheus. That's incorrect. It's the other way around. Prometheus is evaluating the rules and sending alerts to AlertManager which, in turn, is forwarding them to final destinations like Slack, email, etc. What do you use for monitoring, logging, and alerting? What's your favorite stack?

[@oftheriverinthenight]

Prometheus, Loki, alert manager, grafana, slack

[@sriveralopez]

pin this comment!! great video, loving it

[@coocoobau]

I highly recommend kube-prometheus-stack, all-in-one helm chart to deploy prometheus, grafana and alertmanager, each with its own operators. So instead of pre-defining things in values.yaml, you can use CRs to define targets, rules, alerts, dashboards, datasources, etc - in a Kubernetes way. For the logging part, I found banzaiclud's logging-operator to be very interesting, again a way to simplify the deployment of software for logging collection, aggregation and shipment (Loki being just one possible destination). It is also built around an operator and deploys instances of fluentd and fluentbit.

[@nas1k]

I think we should include Tracing here. It can be Jaeger, Temp or something else. And all those thing should be standardized by OpenTelemetry.

[@DevOpsToolkit]

You're right. I should have added tracing to that video. I'll work on making a follow up with tracing

[@DevOpsToolkit]

Here's Jaeger: kzbin.info/www/bejne/fHyTpptjbNN3ick

[@luismorteo6112]

as ever you are rigth, i try loki and wooow! woks perfect with grafana thanks a lot genius!

[@soubinan]

Great video!!! Observability is so important and allow a lot off evolution not yet explored today I have the mnemonic word AMLET for alerting, monitoring, logging, eventing (context and others) and tracing I think also that grafana is the de facto place to have all data to observe even as a saas) thanks to tempo and loki we can add more meaning to metrics dashboards (and I have a small preference for sensu go over robusta to serve as a glue around all that) and leverage all that with a runbook system for auto remediation (stackstorm, awx, ansible platform, jenkins, rundeck....). The dream!

[@RaviSharma-vw7py]

Thanks Viktor for your nice video & informative , really helpful

[@azerbaijan50]

Nice explanation! Thank you very much.

[@fpvclub7256]

Fantastic!

[@daivol666]

It would be interesting to see an example using opentelemetry to gather the observability data (avoding agents vendor lock-in) and use the otel pipelines to expose the data to different vendor solutions.

[@DevOpsToolkit]

Adding it to my TODO list... :)

[@vMILYAv]

VictoriaMetrics (Operator)

[@onemanops]

Yes!

[@MusheghDavtyan]

very interesting

[@andriespiitso9797]

I suggest you also add the alert example just like you do with querying. Otherwise great video enjoyed it 👍🏽

[@jirityr]

If you are serious about monitoring, you need to setup your own monitoring system even on managed kubernetes like EKS, GKE and AKS. I hope you will take the topic of monitoring further with introduction of Prometheus Operator, Grafana Cloud Agent (and GCA Operator), Grafana Operator and perhaps also Grafana Tempo. I would also love to see separate video about VictoriaMetrics that is much better than Prometheus itself.

[@richarmunicosamaniego8216]

MELT stack = Monitoring, Event (alerting +OnCall), Logging and Tracing

[@jemag]

Would be interesting to have a deeper dive, things like Thanos, Tempo, Mimir, etc. Also, what do you think of using their jsonnet libraries to manage those? I found the community helm charts to be not that well maintained and jsonnet is actually pretty flexible for an enterprise setup

[@DevOpsToolkit]

I have Tanos on my TODO list. Adding tempo and Mimir as well... :)

[@claytoncastro2734]

I second that request for tempo. Great video as always viktor.

[@ZachLanich]

You're awesome

[@DevOpsToolkit]

Thanks a ton Zach

[@oftheriverinthenight]

In the latest versions grafana also shows the alert manager alerts and can be silenced from there too (bell icon)

[@DevOpsToolkit]

I was not aware of that. That's great news.

[@gaetanbloch7119]

Thanks!

[@DevOpsToolkit]

Thanks a ton!

[@gaetanbloch7119]

@@DevOpsToolkit My pleasure! Can’t help but contributing a bit as I’m binge watching your videos. Plus, it was a pleasure collaborating with you on Geekle’s conferences. Keep up the great work!

[@vn7057]

After few mo later Grafana stack extended make more flexible Grafana tempo + open telemetry for auto instrument + Grafana agent Grafana loki Prometheus Grafana Alertmanger Basically included Metrics , log, apm/tracing and alert Also Grafana able to adding silence by UI so we don’t need expose Prometheus alertmanger to make alert mute

[@SerhiiHromov]

How did I miss this video, 2 days wasted. Thanks

[@lhxperimental]

This could not have come at a better time! Looking forward to part2 with tracing, open telemetry etc. and maybe also cover the maintenance aspects. Prometheus does automatic data purging which makes it maintenance free; how does loki compare with it. With logs the data volumes are going larger and much more workload dependent so one could easily overwhelm the system. Plus some organizations may need log archives to be kept for several years, how loki supports that use case would be interesting to see. My organization uses elastic search. Can loki be a replacement for elastic search today, or in future? The reason I would prefer loki over elastic is because I can co-relate logs with metrics, events and maybe even traces. In case of java/spring boot based apps, tracing can be very simple to achieve with auto-instrumentation. This would provide great visibility into the working of the application. I am myself exploring it this week.

[@DevOpsToolkit]

Adding those to my TODO list... :) In the meantime, you might want to join kzbin.info/www/bejne/gKLIaJ6hjb-tedE and ask those questions there as well.

[@lhxperimental]

@@DevOpsToolkit Sure, added a reminder

[@DevOpsToolkit]

OpenTelemetry is finally finished and available at kzbin.info/www/bejne/pZaYioyebtKbsNk. Tracing is coming next.

[@dmsalomon]

Great stuff, but the in my opinion the really tricky part is managing these things at scale. First of all there is the storage aspect, but also Prometheus seems to breakdown when the cluster gets too big. At that point you either need to use a federated setup or something else and it would be useful to hear your thoughts on that.

[@DevOpsToolkit]

You're right. When running at scale, a single Prometheus does not work (it cannot scale). I already have it on my TODO list to tackle that subject in one of the upcoming videos.

[@dmsalomon]

@@DevOpsToolkit amazing! Very much looking forward to that

[@joebowbeer]

I concur with others that Tracing is conspicuously absent, as is OpenTelemetry (OTEL), which is the emerging standard that ties all these CNCF pieces together with others such as Fluent Bit

[@DevOpsToolkit]

You're right. I got complaints in the past that the videos were too long so I started making shorter ones and that often results in things missing. I'll work on a second part of that video with tracing and open. telemetry

[@DevOpsToolkit]

I finally got around making a video about OpenTelemetry (I wanted to explore it separately first). It's available at kzbin.info/www/bejne/pZaYioyebtKbsNk. Tracing is coming next.

[@AhmedAyman-gs7oz]

Great video. Are there any other self-managed logging solutions other than ELK/EFK and Loki-Grafana?

[@DevOpsToolkit]

There's fluentd and fluentbit for shipping logs.

[@AhmedAyman-gs7oz]

@@DevOpsToolkit Thanks. I mean more on the storage side. Like if I want to move away from AWS open search to decrease cost. Are there any alternatives than Loki?

[@DevOpsToolkit]

@AhmedAyman-gs7oz Loki is the only one I used besides elasticsearch (excluding managed solutions).

[@samehammar8062]

Thanks for this great video , what about black-box exporter ?

[@DevOpsToolkit]

I should probably create a separate video about a selection of Prometheus exporters.

[@samehammar8062]

@@DevOpsToolkit this would be nice 😊. Thank you 🙏

[@jgarfield]

How do we configure SSO for Grafana login?

[@DevOpsToolkit]

As far as I know, there is no SSO in Grafana open-source version (not sure about Enterprise). I would probably try something like Teleport (you'll find a video in this channel).

[@vn7057]

Grafana is a great tools but somehow like Jenkins If the plug-in no longer maintain then you may get trouble it have chance may your Grafana unable to perform upgrade otherwise graph will dead But I do love the trend they go for Loki+Tempo+Display Prometheus Love to see if there have tools to generate dashboard and integrated 3 of them Do you have plan or already have video for Tempo?

[@DevOpsToolkit]

Tempo is on my todo list :)

[@student_voice]

Thnks man. I got error LogfmtErr. How to solved it?

[@DevOpsToolkit]

Not sure without taking a closer look at what you have...

[@student_voice]

Can you share your email, so that i can share screenshot.?

[@student_voice]

It is showing as : 👇 __error__ : LogfmtParserErr

[@DevOpsToolkit]

Please send me a dm on Twitter or LinkedIn. You'll find my info in a description of any video.