[ Kube 100 ] Getting started with Grafana Loki in Kubernetes

In this video, I will show you how to deploy Loki stack in a Kubernetes cluster and start collecting logs in a central place. The stack will container Loki as the logs storage engine, Promtail as the logs collection agent and Grafana for visualization.
Loki Documentation:
grafana.com/docs/loki/latest/...
LogQL:
grafana.com/docs/loki/latest/...
📺 Learn Kubernetes Playlist:
• Learn Kubernetes
Hope you enjoyed this video. Please share it with your friends and don't forget to subscribe to my channel. For any questions/issues/feedback, please leave me a comment and I will be happy to help.
👏 Thanks for watching.
💗 If you wish to support me:
www.paypal.com/cgi-bin/webscr...

[@leopoldobarreiro9766]

Venkat always bringing great tech contents. Thanks a lot.

[@justmeandopensource]

Hi Leo, many thanks for watching.

[@giarc1988]

This is an amazing tutorial. Bravo and thanks

[@justmeandopensource]

Hi Craig, Thanks for watching.

[@BemusedSoliloquy]

You are a very sharp cookie, I was impressed not even one minute into the video. Your tip of saving out the chart values for reference has accelerated my progress immensely, I don't know why but I had gravitated towards using --set to get things done, no more, tyvm :D

[@justmeandopensource]

Hi Derek, thanks for watching. I am used to using values file :)

[@JohnMatthew1]

Thank you so much for your video, it really helped me get Loki running with my Prometheus installation!

[@justmeandopensource]

Hi John, thanks for watching.

[@teknikalyfit]

With this video, I have learned a huge amount of valuable information. Thanks bro.

[@justmeandopensource]

Glad it helped. Thanks for watching. Cheers.

[@mr.cybernetyc7292]

You knew it :)) I was searching for this info regarding loki , thnx man , much appreciated it.

[@justmeandopensource]

You are welcome and thanks for watching.

[@kamrulahsan06]

Thanks a lot Venkat. Much needed.

[@justmeandopensource]

Hi Kamrul, thanks for watching.

[@danielmarques433]

Thank you. I always learn a lot from you.

[@justmeandopensource]

My pleasure!

[@kevinyu9934]

Thanks for the great content. Really appreciated it.

[@justmeandopensource]

Thanks for watching Chinglong.

[@cezaralmeida1564]

Thanks for this great content! Congrats!

[@justmeandopensource]

Hi Cezar, thanks for watching.

[@MrKumargopalan]

Hi Venkat, excellent video. It was very helpful. Keep doing more videos.

[@justmeandopensource]

Hi, Thanks for watching.

[@chandunallani6283]

Superb content and straight forward

[@justmeandopensource]

Hi Chandu, thanks for watching. CHeers.

[@nacholibre4530]

great content as usual, thanks!

[@justmeandopensource]

Thanks for watching.

[@dannykopping]

Loki maintainer here, great job on the video!

[@justmeandopensource]

Cool. Thanks for watching.

[@zubairhaque2706]

i agree, its good@@justmeandopensource

[@Tikhon_FaiR]

Thank you for the guide!)

[@justmeandopensource]

Hi Tokhon, thanks for watching.

[@user-hp3hr7jf1n]

Thank you very much 👍

[@justmeandopensource]

hi, thanks for watching.

[@julianomoraisbarbosa]

Cool !!! Content of great relevance !!!

[@justmeandopensource]

Thanks

[@suchithreddy733]

Thanks a lot. You saved my day :)

[@justmeandopensource]

Glad it was helpful. Thanks for watching.

[@user-nr8rt1xc2e]

great!! many thanks to you!

[@justmeandopensource]

You are welcome. Thanks for watching.

[@fuliaamiya]

Very straight and specific to solution but details. It is really helpful.

[@justmeandopensource]

Thanks for watching.

[@kumareswaramoorthi8828]

Life saver, Thanks a lot

[@justmeandopensource]

Hi kumar, Thanks for watching.

[@kanwalnainsingh]

Fantastic video.

[@justmeandopensource]

Thanks for watching.

[@networkfreddy2000]

Nice!

[@gouterelo]

Great video Venkat ! Im trying to send to a external grafana the loki service ! thanks again for all the videos !

[@justmeandopensource]

Thanks for watching. You can just add the lok datasource in your external Grafana provided you expose loki outside the cluster and that grafana can reach it. Cheers.

[@liorco300]

Thanks great video

[@justmeandopensource]

Hi, thanks for watching.

[@coppyphommin9337]

Oh Thank you.

[@justmeandopensource]

Thanks for watching. CHeers.

[@Thirumalesh100]

Great content, I could able to deploy Loki for our use case by referring this video. Thanks . I am also looking out the way to get the logs using Loki API for given pod

[@javisartdesign]

awesome thanks!

[@justmeandopensource]

Thanks for watching Javier.

[@skycomandor]

you the best

[@justmeandopensource]

Thanks for watching. Cheers.

[@Pyrusmalus]

Thanks 😊

[@justmeandopensource]

Thanks for watching Adil

[@devmrtcbk]

thank you Venkat

[@justmeandopensource]

Rhanks for watching Murat.

[@aaslanli3943]

Thanks for this good jobs

[@justmeandopensource]

Thanks for watching.

[@guillermobarroso1139]

Nice

[@justmeandopensource]

Thanks for watching

[@pawansolanki20095]

Hi , Venkat . Very informatic video. Which logging soultion you will suggest in production for cluster having more than 50 nodes?

[@dattak2625]

Thank you for all your wonderful videos, I have a query regarding logs, can we use loki to monitor application specific logs, if not would you suggest any tools for application specific? Thank you.

[@himansusekharmishra5114]

Thank you for this amazing video. I have a question that I have a customize storage class of type gp2 along with PV and PVC , so shall I need to mention the PV and PVC in the Loki yaml file or the storage class name will be sufficient enough for the yaml.

[@rmnobarra]

Another amazing video, Mr Venkat. Would be awesome a traefik v2 video.

[@justmeandopensource]

I have already prepared for Traefik v2. Just need to record it. Will do it soon. Cheers.

[@rmnobarra]

@@justmeandopensource Venkat, The only one!! :)

[@yashshah111]

Great tutorial! Is there any way you could make an updated tutorial on this? Can you also touch upon their Grafana Cloud offering versus using the open source community version?

[@rc3043]

This explanation was awesome! I saw your logs have also this prefixed metadata: `stderr` before the real log the JSON. How can you remove this so loki will recognize the json properties correctly!? If you use for example fluent bit instead of Promtail?

[@bharathmc7964]

It's very helpful and to the point. Thanks. If possible, can you please make a small demo of "Simple scalable deployment of Grafana Loki ", which is latest and as loki-stack is mostly depreciated and could not find the official documentation as well.

[@justmeandopensource]

Hi, thanks for watching. Why do you think loki-stack is deprecated? The chart is still maintained in grafana helm repository. I can certainly do a video on simple scalable deployment. I will try and record in the coming weeks. Cheers.

[@sumanniroula3497]

Thanks for the video. I followed the steps but 'Explore' page is empty for me even though loki data source is added. While checking the Javascript console, I see "Datasource named Prometheus was not found". Can you help?

[@sunnynehar]

Hi Venkat, your videos are just mind-blowing, want to ask about any upcoming videos on Grafan Tempo?

[@justmeandopensource]

Hi Gordis, many thanks for watching. I haven't explored Grafana Tempo. I can add it to my list and I am going through my user requested videos list at the moment.

[@wva5089]

Thanks. (hopefully this comment shows interaction with your vids and gets you more views)

[@justmeandopensource]

Thanks 😊

[@Ajmalkhalil-cx4gf]

Hello, I have setup the grafana/loki stack similar way you did. Now in grafana dashboards i am trying to create dashboard for individual pods and i am able to do it. But when i am restarting the pods, pods name gets changed. Now i am not getting any logs on dashboard. I am not sure if its the correct way of setting the dashboard i.e individual dashboards. or i just need to setup logs for namespaces only? would be grateful for your help

[@nikhilpatel4278]

That was awesome! Thank You. Can you please also make some video series on securing Kube API servers with Oauth & OIDC.

[@justmeandopensource]

I have that in my list. Let me see if I get some time to explore that. Cheers.

[@sandeepmishal2376]

Hi Venkat.. It would great if you do for both in house cluster as well as cloud provided managed k8 cluster where one don't have access to the control plane.. Once again it's a great work Thanks again

[@justmeandopensource]

@@sandeepmishal2376 Thanks for watching. I mostly cover the concept in a more generic way and let people explore further. I know people are not going to be running clusters on their laptop in the real world. But I won't be able to cover all the scenarios. Some might be interested in gke and other eks etc. But I will keep this in mind. Cheers.

[@sandeepmishal2376]

Thanks Venkat.. I understood.. May be I could have been more specific to my comment and not confused you.. I was stating it for SSO using oidc topic where it can become a bit tricky as we will not have access to control plane . Sry for the confusion I love the generic way it self..

[@justmeandopensource]

@@sandeepmishal2376 Ah okay. Now it makes sense as to why you replied in this thread. Cheers.

[@user-cd9tj2tw1z]

up!

[@evgeniyburovnikov656]

Thank you so much for the video, it was very helpful for me. I would also be very grateful if someone could tell me how to get the loki, promtoil configuration files after installation???

[@ehtishamrajput3597]

I’ve grafana loki stack running on my local docker desktop k8s , i’m integrating it with GitHub Oauth, what root url and callback url should i set, I’m getting mismatch redirect

[@carmilocabigon6233]

Hi, can you do a video on how to add additional config to get logs for a custom application? What configs do we need to edit, is it the configmap for the created promtail service?

[@MIRISU2]

Hi! Many thanks for that video! I have one question. I've installed loki and grafana. But I don't see how to expand log line and find out more details of log item.

[@justmeandopensource]

Thanks for watching. Do you mean in Grafana that you can't see the full log item? I am not sure if loki truncates the log item. I would be surprised if it does that and there is no configuration to control that behaviour.

[@HDGlazaa]

Can you tell me how to change config file for loki ? for example set max_query_series = 1000 ( dafault is 500 ) in kubernetes

[@kranteshshrestha361]

Hello can you please make a video for setting up log retention period in loki stack. As i have tried to set the parameter according to documentation but doesn't retain the log according to set period.

[@ChristianAltamiranoAyala]

Nice tutorial, Did you test the query performace with a large mount of data? If so, Is loki more performant than EFK?

[@justmeandopensource]

Hi Chris, thanks for watching. I haven't tested the performance with large dataset. But according to them, it should perform better than Elasticsearch as it doesn't have the index overhead.

[@tamilan_editzz]

Hey hi, actually im using xk6 for sending logs to loki, is it possible to send our own messages to loki like " This is loki test by chandru" Message?

[@Sony-il2ti]

deploy Loki in my cluster using helm but promtail was not deployed to the bare metal servers, how do i get it deployed on them too?

[@mahfoudissam1285]

hi thanks for this guide,but how can i display all the namespaces of the cluster,because loki display only some namespaces to me in the grafana dashboard

[@mahfoudissam1285]

hi thanks you ,i have found the answer , there was any logs to display in the other namespaces

[@dattak2625]

Hi, I am getting 502 bad gateway error, when I test data source in grafana, can you help me with this, my setup is with kubeadm on aws and it's single node, I am accessing grafana with nodeport.

[@student_voice]

Hey i have setup a Dashboard for App logs in Grafana - Loki. Logs are coming but there is an error showing "LogfmtParserErr" The actual error looks like this: __error__ : LogfmtParserErr

[@Dheerajpant5211]

Thanks for the explanation, is there any way that grafana can send full information of logs in email as alert, we are getting alert but it is only giving count, if there any way to get full information of log in email?

[@justmeandopensource]

Hi, thanks for watching. You could use alert manager and configure it to your liking.

[@manidevops8501]

Thanks for such a wonderful video. I'll really enjoyed the way explained. Can we do log rotation in LOKI? bcoz, in my production environment the EBS volume is getting filled with lots of logs. if i want to delete the logs which is 10 days older, how will i do that with loki?

[@justmeandopensource]

Thanks for watching. You have to configure retention settings in storage configuration. grafana.com/docs/loki/latest/operations/storage/retention/

[@manidevops8501]

@@justmeandopensource thank you so much..........

[@justmeandopensource]

You are welcome.

[@vikasp1262]

Thanks a lot, how can i make alerts if any pod having error in logs

[@justmeandopensource]

Thanks for watching. You have to use alertmanager for that I guess. Haven't tried it myself.

[@bunnyzero9125]

Hi can anyone tell me how to get MySQL server query logs to Loki grafana dashboard, Mysql query logs are in pod container (opt/bitnami/MySQL/logs/query.log ) i want this logs which are in this pod container location to show displayed in Loki grafana dashboard can anyone here guide through this

[@bharatarya7929]

Hi! How to handle archiving of logs

[@pjj7466]

Bro can you please make another video with latest loki monolithic setup and microservices loki setup on kubernetes please?

[@2mahender]

thank you, can you let me know how to setup alerts for loki stack

[@justmeandopensource]

Hi, Thanks for watching. I haven't tried it myself but here is the documentation that might be helpful.

[@bmanikantareddy]

what is the browser you are using for demos. I can see some tabs and running commands for opening the URLs

[@justmeandopensource]

I use Qutebrowser as my web browser on Arch Linux. This browser is python based and keyboard driven. You will find it extremely useful if you are a keyboard centric person and VIM user.

[@justmeandopensource]

www.qutebrowser.org/

[@malcolmserrao8721]

nice explanation, i have a question, if i need to configure logging on multiple kubernetes clusters can i use a single grafana instance for viewing the loki logs from different clusters?, appreciate your help

[@justmeandopensource]

Hi Malcolm, thanks for watching. Yes that set up is entirely possible. You can have all logging stuff (like prometheus, grafana etc) on a separate cluster and configure ingress routes so that they can be connected from agents on other clusters.

[@malcolmserrao8721]

@@justmeandopensource thank you very much for the quick reply

[@justmeandopensource]

@@malcolmserrao8721 no worries

[@soukhyabhat9588]

Hi Venkat,Can you Please give me yaml file for adding S3 bucket as storage .Have tried many ways but nothing is working .It would be great if you could help

[@mzw8374]

When tried to setup data source for loki, it was unable to connect to loki service. Can you please help for solve this?

[@pavankumarnalajala2678]

Please make videos on Splunk for monitoring of k8s.

[@justmeandopensource]

I can give it a try

[@shamanthreddy8154]

Can You do a fluentd with loki and Amazon managed grafana

[@pisckipratama9529]

Hi Venkat, thank you for informatic video. I'm deploying it with grafana disabled, and how do I update the grafana enabled without deleting the configured?

[@justmeandopensource]

Hi Piscki, thanks for watching. I am not sure if I understood your question correctly. Can you elaborate your requirement please? Thanks.

[@pisckipratama9529]

@@justmeandopensource so i have deployed loki-stack with grafana disabled, but i want to change it to enable, is it possible without running "helmet uninstall" command?

[@justmeandopensource]

@@pisckipratama9529 Ah okay. You should be able to change most of the stuff without uninstalling. For example, as shown in this video, if you checked out the values of the chart with "helm show values" to a local file and used that in helm install command with --values option, then you can edit that values file, enable grafana and run "helm upgrade --install --values ". I haven't tried it in this case but I know it worked for me in the past. However there are certain stuff in the values file that you can update on live deployment. You will have to helm uninstall it first and deploy again.

[@pisckipratama9529]

@@justmeandopensource ohh okay Venkat, thanks for the good explanation

[@justmeandopensource]

You are welcome.

[@amarchand6981]

Which shell are you using?

[@justmeandopensource]

Zsh along with some goodies which I have explained in this video if you would like to watch. kzbin.info/www/bejne/hoa6n3aYp56WhJo

[@Ariel-om1lt]

requesting for Grafana tempo too :)

[@justmeandopensource]

I will add it to my list. Cheers.

[@Ariel-om1lt]

@@justmeandopensource Thank you! as always great content!

[@justmeandopensource]

@@Ariel-om1lt You are welcome.

[@sealedd]

No need to setup a PersistentVolumeClaim?

[@justmeandopensource]

It all comes from helm chart. No need to setup pvc but you should have automated volume provisioning in your cluster.

[@user-kt9gr4iu9h]

how to download these logs

[@justmeandopensource]

Hi, thanks for watching. I am not sure why you would want to download the logs. Its because you wanted to retain the logs from various applications, you have deployed Loki and storing logs in there where you can query and visualize stuff. If you want to download logs, you will have to write a query and store the output in a file. Hope this helps.

[@YongUnHeo]

good job man! i want use nodeport service on loki-stack-grafana. teach me edit option. tmp/loki-stack-values.yaml

[@justmeandopensource]

Thanks for watching. I will try it out and if possible do a follow up video to this one. Cheers.

[@bobbyhamel4345]

I don't understand why you would skip setting up an ingress, in a real production env you wouldn't use port forwarding... All tutorials related to Loki fail to acknowledge the ingress.

[@ervikasrastogi]

Hi....I have loki promtail and grafana setup and want to see grafana usage per user with dashboard from grafana.log..I have referer label in logs which is having dashboard name along with https url but i only want to show dashboards accessed userwise..can you please help me for same?