Fluentd on Kubernetes: Log collection explained

Subscribe to show your support! goo.gl/1Ty1Q2 .
Patreon 👉🏽 / marceldempers
In this video we take a look at log collection on a kubernetes cluster.
We take a look at the fluentd dockerfile, how to run it on each kubernetes node using a daemonset.
More importantly is the configuration required to tell fluentd how to collect logs. We also look into where logs are stored on each kubernetes node.
Finally we see how to ship the collected logs out of the cluster.
Checkout my Introduction to Logging video:
• Understanding Logging:...
Checkout my Introduction to Fluentd video:
• Introduction to Fluent...
Checkout the source code below 👇🏽 and follow along 🤓
Also if you want to support the channel further, become a member 😎
marceldempers....
Checkout "That DevOps Community" too
marceldempers....
Source Code 🧐
--------------------------------------------------------------
github.com/mar...
If you are new to Kubernetes, check out my getting started playlist on Kubernetes below :)
Kubernetes Guide for Beginners:
---------------------------------------------------
• Kubernetes development...
Kubernetes Monitoring Guide:
-----------------------------------------------
• Kubernetes Monitoring ...
Kubernetes Secret Management Guide:
--------------------------------------------------------------
• Kubernetes Secret Mana...
Like and Subscribe for more :)
Follow me on socials!
marceldempers.dev
Twitter | / marceldempers
GitHub | github.com/mar...
Facebook | thatdevop...
LinkedIn | / marceldempers
Instagram | / thatdevopsguy
Music:
Track: Amine Maxwell - Amine Maxwell & MICHAL - Chill Night | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / amine-maxwell-michal-c...
Track: POLAROID PAPI - audi | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / audi
Track: Amine Maxwell - Amine Maxwell & Soyb - You & I | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / amine-maxwell-soyb-you-i
Track: Amine Maxwell - Night And The City | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / night-and-the-city
Track: LiQWYD - Bridges (Free download) | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / bridges-free-download
Track: Soyb - Soyb & Amine Maxwell - California | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / soyb-amine-maxwell-cal...
Track: souKo - souKo - Parallel | is licensed under a Creative Commons Attribution licence (creativecommon...)
Listen: / parallel

[@amukher1]

This is perhaps the best single resource to understand how to setup fluentd (and by extension Fluent Bit) for a k8s environment. There's so much of crud out there in the form of blog articles and documentation, not to mention a book by a certain well-known publisher, all of which are completely useless. It really should be a lot simpler - this video takes care of it.

[@videobrice]

Thanks for this excellent and exhaustive tutorial !

[@MarcelDempers]

Thanks so much for the kind words and support 🙏🏼

[@videomaster1393]

Bro, trust me you are "that guy"! every minute is valuable and go straight to the point!! Big thanks!!

[@Jak6711]

I've got to say, I really appreciate these videos. As an intern trying to get into the field of DevOps, your tutorials are always my go to. Fast, concise and extremely helpful. Thanks alot!

[@sravankumar..]

You are the one stop to clear all my questions on kuberentes. Just I need to watch all your kubernetes videos to clear all my questions, great man 🙏🏽🙏🏽 You are a wonderful guy, love from India.

[@devops-techstack]

Thank you so much for sharing this knowledge . It helped a lot while i was struggling to send logs to Stackdriver.

[@bisuaws5377]

You are the genuine Devops Builder, many thanks 😊

[@Misanthrope84]

Another brilliant video mate 🤟 please keep them coming!

[@ckalas]

new video idea: the implications of k8s dropping docker as default container runtime

[@oraculotube]

Thank you very much to share your knowledge. much appreciate that.

[@ThiagoSilva-87]

Thanks for the teaching. You are amazing !!!

[@arpanchakraborty9874]

Amazing, thanks so much, very simply explained yet very informative and almost all you need to know to start off with EFK.

[@krishnakanth6106]

Very well explained.... Thanks a lot😊

[@rabithbo7089]

Thanks so much, it is amazing tutorial.

[@vijaytamhankar5874]

Nicely explained. Easy to follow steps 👍

[@rahulsingh-iq4gd]

Sheer knowledge… only , thank you so much

[@roninl4348]

Thanks a lot man.

[@vladimirsiman8838]

Cool stuff, thanks a lot ! Great !

[@heyitsmekishore]

Thanks for the Video it's really helpful. Can you please tell how to fetch logs from one particular namespace

[@brenokevin3995]

Very good, I really liked the video :D

[@1006931080]

You are doing a great job buddy, thanks for awesome video. You have any plans for fluentbit?

[@shahrukh49]

Very Helpful. Thanks.

[@LuisMiguelZapata]

Amazing.. Thank you!

[@lucaricchi5873]

Loved this! What kind of english accent is this? As an italian speaker I can not guess ;) great job btw

[@irfannian1502]

For a weird reason, I am unable to find file-test.log in my /tmp/ folder as shown in 20:45. Prior to that, everything seems to be executing fine.

[@mula.nareshgowd1381]

I am also caught up with error.could you please let me know how to resolve this @irfan Nian

[@ankitlakum1]

fluent and magical

[@DevOpsKey1]

Nice mentor

[@amukher1]

A basic question. How does Fluent Bit enrich the logs with kubernetes container, pod, and namespace information? Does it derive it from file naming conventions (and then from kube-api as you mentioned)?

[@learndevopswithadnan9569]

Thanks for sharing the beautifully explained Fluentd-elasticsearch as K8S daemonset. I am using ES on https with SSL implementation and facing "error_class=Fluent::Plugin::ElasticsearchErrorHandler::ElasticsearchError error="400 - Rejected by Elasticsearch" location=nil ..." Any hint regarding this error.

[@UnknownSend3r]

Did you find the reason you were getting such an error ?

[@PePTo-dx2yj]

may be somebody recive this, Im using containerd instead docker (CRI) in k8s, when was docker all works fine but containerd : [warn]: #0 [in_tail_container_logs] pattern not matched ....what can be trouble?

[@anandnandu968]

sir i want to deploy only fluentd in k8s cluster without kibana and elasticksearch . is it possible.

[@PankajPandey-pw4wp]

Thanks for this video. what is the best namespace to run daemon-sets like node exporter or fluentd ? namespace where application pods are already running or "kube-system" or completely separate name space?

[@MarcelDempers]

It's a personal choice I prefer to leave Kubernetes stuff in kube-system and introduce my own purpose-built namespaces like "monitoring" for all monitoring related things. I would also keep these things far away from application namespaces 💪🏽

[@PankajPandey-pw4wp]

@@MarcelDempers thanks

[@aditiarahman435]

hello I have my logs inserted to elasticsearch from kubernetes pods, I try direct http query to elasticssearch and all the data logs are shown, but some logs with long string are not showing in kibana, do you know what causing this?

[@iimnurdiansyah53]

Hello, what is the big different between fluentd and fluentbit? When we choose fluentd or fluentbit?

[@user-sv9us4rh3v]

Is it possible to run it as sidecar container ?

[@bhuvanparashar1632]

Hi, what will happen if I use more than one source targeting the same path?

[@tiny3241]

thank you! can I direct custom logs to write to fluentd ? like some log file in /opt path

[@MarcelDempers]

For k8s , this is an anti pattern, as fluentd reads logs from the container runtime like docker. I'd get DEVs to start writing logs to stdout instead. For legacy apps , you could have a sidecar app that reads the file and writes to stdout for fluentd to pick up.

[@georgelza]

... another request... any chance you can do a video of doing all of this on a minikube environment, as probably used by allot as a on laptop dev environment. ... as we're on a laptop we prob won't have a local ELK stack, so dumping all likes into a single text file is nice as you can then locally only tail one file, and see the log messages from all the pods running. :)

[@DevOpsKey1]

Please genrate a Video on fluentd and prometheus configuration in K8s using yml file without helm.

[@rahulchauhan3188]

I have multiple namespaces, how can i get logs from all the pods present is particular single namespace? please reply asap, it kind of urgency sir. thanks and love ❤️ from india.

[@nitishsharma5814]

Nice content. Thanks. Can you please help us in using the kubernetes _metadata as the file_name or indexname for elasticearch

[@1milad]

hi, how can store logs per namespace in different index?

[@gnaneethi81]

Amazing , How do we get appended the index_name fluentd-k8s fluentd-k8s-appname

[@csgrinds]

Great video - btw is pronounced demon!

[@nikhuge6580]

Great

[@deepaknayak9912]

Hi, Can Fluentd replace #autodiscover feature of filebeat? Which helps to get the logs of #autoscaling environment

[@emiribrahimbegovic813]

how do you deal with log rotation?

[@georgelza]

Hi Marcel, any idea when you will be able to do a ElasticSearch stack for us ? ... using the Beats framework.

[@MarcelDempers]

I'm not 100% on an ETA, but planning an Elastic Search series at some point, similar to the RabbitMQ + Redis series 💪🏽

[@georgelza]

@@MarcelDempers would be nice if its like the others, ElasticSearch+Logstash+Kibana on Kind, data coming from basic OS via FileBeat, maybe some metrics via MetricBeat and as per always instrument a K8S cluster... if I can try my luck. hope the "point" is soon, hehehehe

[@georgelza]

@@MarcelDempers ... got idea how to make it a tad more complicated, have *Beats push data onto a Kafka Topic, and have Kafka then sink it via a connector directly into ElasticSearch, bypassing Logstash. The aggregate and other math can be done on Kafka via kSQLDB and stream processing... now thats really pushing my luck, hehehehe

[@Ashe94]

So this example only collects stdout, right? If I want to have apache errors logs or something I need to include them in my fluentd Config?

[@MarcelDempers]

In Kubernetes you're better off configuring Apache container to write to stdout instead of a log file. this makes container log collection transparent across the system and more scalable

[@hellboy0666006660]

@@MarcelDempers Would we then write error logs and access logs to stdout? Usually they are separated in different files (e.g. nginx), but if I write both to stdout they will be mixed in one 'file'. Would that be ok, to not have that separated?

[@MarcelDempers]

@@hellboy0666006660 That would be ok. They may go out of the system in the same stream, but you'll be able to view both in what ever system you are analyzing them in. I.E in elastic search is where you will be able to separate them

[@elidrissii]

@@hellboy0666006660 I personally write error logs to stderr. It's just cleaner to keep the separation.

[@abhichennupati16]

I see my pos_file is missing, what might be the reason?

[@MarcelDempers]

Make sure you retain it somewhere, like a volume mount when using a container. Else it would be lost when the container restarts

[@muralikrishnaallaparthi2480]

@@MarcelDempers sir, what lines to be included in the yaml file for volume mount of the pos_file. One more question, In this vedio we are writing the logs to elastic search or output file. Once the log is written to elastic search whether it is deleted automatically from pods.

[@bombdrive3880]

kubernetes_metadata could only enrich data from /var/log/containers. It claims to be able to enrich "application data", but it actually does not... = =. The documentation is so misleading.

[@vibinck19]

100 likes