How Fluentd simplifies collecting and consuming logs | Fluentd simply explained

[@TechWorldwithNana]

► Follow me on IG for behind the scenes content 😊👉🏼 bit.ly/2F3LXYJ ► This video was sponsored by the Cloud Native Computing Foundation www.cncf.io/ 🙌🏼 ► More infos on Fluentd: www.fluentd.org/ ▬▬▬▬▬▬ T I M E S T A M P S ⏰ ▬▬▬▬▬▬ 0:00 - Intro 0:26 - Why we actually need log data? 1:40 - Challenges of collecting and consuming logs 3:40 - Solution - a unified logging layer 4:17 - How does Fluentd work 6:29 - How to configure Fluentd? 9:03 - Built-In Reliability 10:09 - Other Use Cases

[@tonytwostep_]

Have had fluentd daemonset set up in my kubernetes cluster, pushing pod, service, and other logs to elasticsearch for a month or so now. Really like the simplicity and flexibility it offers.

[@fhajji]

Never tried fluentd before, but I definitively will, thanks to this video. I love this channel. Please keep 'em coming.

[@TechWorldwithNana]

Glad to hear, thank you Farid :)

[@ayushsinghrathore7186]

That's amazing 🔥 We need more tutorials on EFK stack 🙏

[@RaulNussbaum]

Very interesting. I heard about fluentd in another place and came here to find some info. Glad I found your video, easy to understand and digest. Thanks!

[@rraudales]

Nice intro. I was expecting more of a how to actually install and configure fluentd

[@muelladresse10]

Nice, my attention was drawn to fluentd by your Udemy course and I was looking for more information about it a few days ago and now you made a video about it 👌

[@TechWorldwithNana]

😊👍

[@polmaksim]

Dear Nana. Very interesting Your opinion. Fluentd vs Fluent-bit? And which product is better to use in 2021? Thank you 😊

[@TechWorldwithNana]

I will actually make a follow-up video on Fluentbit as well. One of the common setups, especially in bigger clusters is, fluentbit pods as collectors and fluentd as an aggregator. Because fluentbit is more lightweight, so such setup would give you good performance while saving resources.

[@polmaksim]

@@TechWorldwithNana Thank you Nana!

[@TheVicky7374]

first of all, you are my favourite in my internet world, I learn a lot from you, thank you mam. sorry for my english. in my language which is my favourite आपको बहुत बहुत धन्यवाद।

[@joelamks]

Great. Nana. You always enlightened my knowledge. EFK becomes my target K8s logging framework. Thanks.

[@samuelelard7439]

Thanks for your videos, I'm a pretty bad english listener and I understood all you said and I learned a lot with !!!!

[@VasilyMT]

Nice - short but informative. Thank you.

[@TechWorldwithNana]

Thanks Vasily :)

[@uve_viktor_doom]

wow, super good quality video, thanks for the great effort!

[@arariku7]

Nice explaination. Good job. Can you also do a video on OpenTelemetry?

[@sandeshsonkamble1264]

Another bang on video thank you for your help please add some slides to take notes.

[@anthonyfarias321]

I strugle with this a few days ago. I wish i had found this video earlier! thanks

[@DevOpsJourney]

If I want anyone collecting my data... It's Fluentd!

[@benschipper]

Great Video Nana! I use FluentD for quite some time already collecting logs and sending them to specific indexes outisde of the cluster so they can easially be observed using query's within observability platforms :)

[@nintran52]

Hi Anna, your video is always perfect and helpful. I wonder how you can make like this video, what are tools to do this

[@rouabahoussama]

what about logstash !! thanks for the video .

[@TechWorldwithNana]

Fluentd is actually more fit for K8s, it was actually created with k8s in mind.

[@andrewclark3325]

@@TechWorldwithNana Fluentd was created when such things as Kubernetes does not exist at all

[@OlleHellman]

Thank you for the nice presentation. What tool would you recommend using when visualizing the log data?

[@TechWorldwithNana]

I've used EFK stack for that so I think Kibana is really great for visualising logs!

[@tiktokfans9019]

You are the best woman in the world , I love ur info ❤

[@adrianfgutierrez2836]

Spectacular explanation!!! 👏🏽👏🏽👏🏽👏🏽👏🏽

[@Yourfriend-s7n]

Hey Nana, the content of your video is always very good and easily understandable. Except for your audio level, it seems low to me. I have tried many ways to increase your audio and try different devices as well but it didnt increase the sound volume. please make sure your videos are having little high audio quality so that some one can easily listen to it without earphones or headphones. I hope it wouldn`t appear offensive to you.

[@MVOralov]

I do really like the way you express and deliver your thoughts, much, much appreciated! Please go on with your videos! :) +

[@MVOralov]

BTW - how can we support you, donating anyhow? Thanks!

[@olehigorovich474]

This content is so good it should be paid

[@timothyn4699]

very well broken down, the config snippets and animations help to visualize it a as well, thanks !

[@nhattran536]

The DevOps hero we didn't know we needed :)

[@SanjeevKumar-nq8td]

Thank you. The contents are same as your Fluentbit lecture. Could you please give bit precise use case info when to use which one or how to use both.

[@abdulmohidmd7589]

great explanation as always Nana, what is the logging solution we can integrate for Prometheus to collect application logs?

[@PrakashReddyK]

As usual Absolutely fantastic content thanks a lot for your wonderful explanation.

[@altayhunoglu3552]

A great way of explanation indeed.

[@pratikjain2690]

Filebit or logstash also does same thing then why we should go with fluentd.. And also plz explain fluentd config file in more details..

[@TechWorldwithNana]

That's a common question. Fluentd is more fit for k8s than Logstash, it integrates better and was created with k8s in mind. Plus it has some advantages, like routing the logs is much easier in Fluentd as well as Logstash needs additional service for data persistence. These are some of the main diffs.

[@gnanasekaranebinezar7199]

It's worth watching and learning too. How it differs from Jaeger ? Please throw some light on that. Thanks in advance.

[@CoupleVloggerInAustralia]

Amazing video, thanks a lot🔥.Could you also please make one lesson on using ArgoCD for Kubernetes as the GitOps continuous delivery tool?

[@mrsvvmrsvv7400]

Can fluentd collect logs using push-model like logstash? It is convenient to just add handler/appender to e.g. logback configuration and forget about log format.

[@olge1355]

Very well explained.

[@devanshukhokhani4518]

Could you share your views on how Fluentd is different then splunk for logging apart from the fact that splunk is commercial

[@markpalmer5271]

Thank you ! very useful content

[@TechWorldwithNana]

Thank you Mark!

[@aartavazd]

Dear Nana, I like your videos they are very good with good content. But in this case it was not the quality I was expecting :) your videos are watched by IT pros you need to give more in dept knowledge instead of just high level overview. Please :) But overall I like your vides, keep up the good work :)

[@moustafa6006]

great video 🎉

[@nforlife]

Hi @TechWorldwothNana when the the DevSecOps course going to be available

[@raulalejandromunozaraya3878]

You don’t have patreon or smth like that? You are doing an amazing job helping the rest of the community and I would like to give back smth :)

[@rokofromhell9264]

Thank you so much! Is very interisting

[@saikattorrents]

Thanks for sharing.. Though AWS documents suggesting to use FluentBit instead of FluentD..

[@TechWorldwithNana]

Yes Fluentbit is more lightweight. Needs much less resources. However Fluentd has more features, plugins. So depends on what you need to configure. If Fluentbit functionality is enough, then you can go with it. I will actually make a video about Fluentbit next month :)

[@vikramanand8277]

@@TechWorldwithNana You care for us so much :)

[@01091a0477]

Dear Nana, could you please provide the video on design , how to get the health and pod level metrics for the applications deployed in openshift?

[@devops__guy8909]

should I choose Fluentd over logstash/filebeat? @nana

[@suresha7988]

Thanks for your videos

[@ac1dr3d80]

Great content as always. Thank you!

[@jig1056]

Hello.. thanks for taking the time to put together this video. It’s excellent, it really does help with conceptualizing how to utilize fluentd.. I got it up and running and I am capturing logs from several of my back up processes and syslogs from few servers. Something I’ve not been able to figure out is how to dynamically assign labels to my Loki match blocks. I’ve used the rewrite_tag_filter plugin to push log lines to different match blocks where I have different labels but that feels like I’m hard coding the same thing over and over just to get one label to be different. I’d like to store the program or process name in a variable based on the log line and use it as a label. Is that possible? I’ve tried several different ways but I have not been successful doing that as yet. Any info would be greatly appreciated. Thanks again *edit when I say labels I’m referring to labels used in Loki and visible in Grafana

[@danielkrajnik3817]

Great content (same comment again but this content is really good)

[@TechWorldwithNana]

Happy to hear, thank you Daniel 😊

[@pallavkanaujiya]

As usual the best!!

[@fbmello]

Thank you !!!

[@gangadharrao7186]

very useful

[@shreyshrivastava]

Can this be used for logs from iOS application?

[@milad987]

Nice video, but I can't get the difference between this tool and a broker tool as for example CAMEL

[@gary5743]

Do you have any plans on creating videos about podman, buildah, skopeo?

[@TechWorldwithNana]

I didn't plan it for now, but will definitely consider them! 👍

[@InterNeo69]

Is it worth using fluentd together with Prometheus? Or is it more like an alternative?

[@shijucan1]

I'm new to all this. Is nlog just another logger like fluentd?

[@PankajPandey-pw4wp]

Thanks for this video. is there a way to limit nested parsing in json parser plugin @t ? as current it's parsing json in all levels to lots of field getting created. to wanted to limit josn parsing on first level.

[@abhijitpaul6928]

Thanks for the video...Currently I am using Fluentd but it's can't go beyond 2k/EventPerSec/PerIntance. Where I am looking for 20k/EPS/PerInstance with each average event size of 800 Byte. Do you have any suggestion for the same?

[@Fayaz-Rehman]

Thank you - Crystal clear - ***** Stars

[@TechWorldwithNana]

Thanks Fayaz 🙏

[@just-thinking-straight]

start at 4:22 :)

[@nah0221]

Can Fluentd collect pod/containers metrics ? if not, what do so in Docker & Kubernetes environments ?

[@AlexanderAtlas]

Do you have lessons on linked in as well ? Couldn’t find ?

[@ranjaniramanujam8038]

Can you search/parse collect a specific string in fluentD like sumo logic. Can you set up dashboards on FluentD itself without pushing it to a destination system

[@deepaknayak9912]

Hi Nana, Can Fluentd replace #autodiscover feature of filebeat? Which helps to get the logs of #AUTOSCALING environment

[@Василий-э8ч4ш]

What's the difference with ELK stack?

[@amyqb117]

Need to know this too.

[@Ajitsingh-gc5ej]

Can u please let me know how to push specific namespace logs only to elasticsearch from fluentd in k8s any specific parameters we need to use ?

[@muppiriguna84]

Thanks for the video. is it possible fluent-logger-java integrate in android? if possible please provide details?

[@dpanassol]

It should be somehow marked that this is K8s related instruction. Im looking for plain docker instructions

[@florih91]

Hey how does it compare to cribl ? I cant rely spot wheres the diffences right now.

[@_MD80]

Subbed👍🏻

[@rampanwar1316]

Suberb mam

[@girijaprasannamallick9649]

which one is most fit for gcp cloud run logs collecting?

[@MuhammedDahab]

thanx ❤❤

[@tanagool]

Nana , sound volume is a bit low , :)

[@TechWorldwithNana]

Thanks for the feedback!

[@zhangxiao321]

why not looking at a more capable log management solution such as syslog-ng?

[@nigratruo]

I would say that I don't consume logs, I don't eat them, I don't inhale or inject them into my bloodstream ;-) I normally just view them. I prefer consuming food.

[@gnaneethi81]

How to create Fluentd index with application name , Like Fluend-containername , deployed with helm

[@venkatk8314]

Dear Nana, when i try to deploy fluentd using helm chart by default it deploys as root user. i changed to non root user by using security context. But the problem is it errors out "[error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/fluentd-containers.log.pos". Can you please advise in case if you had this issue

[@glowiever]

still wonder why logstash is left behind and fluentd becomes more common

[@srinivasyadav9925]

Hello.. how is it different from Splunk?

[@lum4707]

You could compare FluentD to Splunk forwarders. Preparsing, transforms etc. It's not a full product as Splunk, just a part of a stack.

[@MrDevZero]

დიდი მადლობა მსგავსი კარგი ტუტორიალები არსად არაა)

[@ashmansukhani3928]

Nana, I need your help,...!!!!

[@ashmansukhani3928]

I am having a bad experience for the last 3 days even though I have paid for the course. Please provide me a refund if you cannot help me open the second video of this course....

[@TechWorldwithNana]

Hi Ash, can you please send an email to [e-mail] and provide email with which you have purchased the course? So we can check. On KZbin comments there is no support Is this your email: [e-mail] ?

[@rahul_bali]

why pointing the finger in thumbnail ? '-[

[@mikekhachadurian4876]

it would be much better if you explained how to install and manage it than listening of 10 minuts of bla bla bla fluentd is good fluentd is the best

[@theaugmenter]

First

[@WildLifeBackyardCamera]

all logs to splunk, period !

[@EduardoSilvaPereira]

many Splunk users use Fluentd to perform the data ingestion to Splunk, there is a good value having the option to control "which" data is ingested.

[@루이스-b9w]

I don't think the explanation is good. it is so abstracted and I wonder if you really have configured it by your self. Is Fluentd flies along the applications like the animation you put on 4:26?