[ ElasticSearch 3 ] How to install EFK stack using Docker with Fluentd

In this video, I will show you how to deploy EFK stack using Docker containers step by step. Also will show you how you can forward system logs from a CentOS 7 client to this EFK stack.
* Elasticsearch - The storage/search engine (www.elastic.co/)
* Fluentd - Log collections and parsing engine (www.fluentd.org/)
* Kibana - Visualization tool (www.elastic.co/products/kibana)
Github: github.com/justmeandopensourc...
Hope you enjoyed this video. If you liked it please share it with your friends. Please don't forget to subscribe to my channel.
Any questions/issues/feedback, please leave me a comment.
Thanks for your time watching this video.
If you wish to support me:
www.paypal.com/cgi-bin/webscr...

[@mukundrangachari5721]

Excellent presentation, confident and straight to the point. Appreciate you taking time to put this together

[@justmeandopensource]

Hi Mukund, many thanks for your comment. Cheers.

[@earlkevinoliverio9994]

I love the flow of your explanation especially indicating which is which with the diagram. It's very easy to understand. Thank you for your hard work.

[@earlkevinoliverio9994]

Is there a reason why the td-agent wasn't containerized?

[@justmeandopensource]

You made my day. Thanks for watching this video and the feedback you provided.

[@justmeandopensource]

Good question about why td-agent wasn't containerized. It can be. But for some reason I decided to go with rpm installation. If you want you can use it as a container. docs.fluentd.org/v0.12/articles/install-by-docker Thanks, Venkat

[@roginrgn89]

Awesome video.. Everything you explained very clearly, waiting for part 2 of efk :)

[@justmeandopensource]

Hi Rogin, thanks for watching this video. Was recorded a while ago. Did I mention anything about part 2? Thanks.

[@JOSCHLI1304SVideosundTutorials]

@@justmeandopensource You've mentioned it at 7:47. Would be great if you would do another video on the EFK Stack. You could also do an updated one.

[@gcppractice8858]

very easy to setup and understand the basics.. thank you Venkatesh for making elk so easy :)

[@justmeandopensource]

Thanks for watching. Cheers.

[@leegaines2391]

Thanks for sharing this video. I haven't used lxc before, but it seems awesome. Gonna try it out :)

[@justmeandopensource]

Hi Lee, thanks for watching.

[@suvankardas7932]

your tutorial is beyond paid tutorial ..so simple and precise..I liked a lot..one small request if you could create a video on 'graylog' would be great..!!!

[@justmeandopensource]

Thank you for watching. I will check graylog and see if I can have a play with it.

[@suvankardas7932]

@@justmeandopensource eagerly waiting

[@justmeandopensource]

I spent some time looking into this. There is no official helm chart available to deploy graylog in kubernetes. Although there is this from kong-z repository. github.com/KongZ/charts. I tried and it didn't work. There were lots of open issues with that chart and it isn't worth spending time looking into it. If I were you, what I would have done is, use mongodb chart from bitnami and elasticsearch chart from helm stable repo and for graylog I would create one myself. Official documentation for graylog has instructions on how to run it in docker. So there is a container image for graylog. You can create helm chart for graylog yourself. I don't think I will have time to look into this further I am afraid. I will give it a try later if time permits. Thank you.

[@gabrielquinteros2101]

Excelent ! thanks from Argentine !!!

[@justmeandopensource]

Hi Gavriel, many thanks for watching. Cheers.

[@user-ye8zh3ep7m]

Nice!

[@justmeandopensource]

Hi, thanks for watching.

[@enissay9950]

Just amazing! Thank you for sharing ♥️ I am curious about a few points : How to setup ssl Is it possible to dockerize the td agent to make it easy to deploy on large number of servers Finally, I am curious what kind of use cases, dashboard,... one could set up in Kiba a (as power user)

[@bondrags]

Hi Venkat, Thanks much really helped out with Fluentd, Appreciate if you could do a video to push the logs from fluentd to Splunk

[@justmeandopensource]

Hi, thanks for watching. I will see if I can do that. Cheers.

[@nfperrini]

Great video... I hope you changed your smoke detector battery though.

[@justmeandopensource]

Haha. BTW Thanks for watching.

[@shishirroy1516]

Excellent tutorial. can we push docker swarm cluster logs to ES using fluentd.

[@ashishpalsingh244]

in my envirnoment we have two types of index pattern . one used for pods on kubernetes and 2nd is used for the FE appliactions .so how will i get to know where i have defined the index pattern name for FE application .

[@Babbar_supreme_leader]

really appreciated, and like accent.. caould you please explain the difference between logstash and fuentd

[@justmeandopensource]

hi Bharat, thanks for watching. Logstash is log parser while fluentd is log shipper. You install fluentd on a client machine and fluentd takes care of collecting and sending the logs to either elasticsearch directly or to a logstash for parsing before being stored in elasticsearch. You can parse the logs and transform/filter in logstash and then forward to elasticsearch. Hope this makes sense. Cheers.

[@_siva_polisetty]

Hi Venkat thanks for the video, I would like to setup elk/efk stack for kubernetes. so, my doubt is about indexes, how to create a indexes and how to send or forward particular logs to particular index, for example I have multiple namespace in k8s cluster and I would like to forward test1 namespace logs to filebeat-namespace-test1* something like that. Could you please help me that thanks in advance.

[@justmeandopensource]

Hi Siva, Thanks for watching this video. I have done a video related to your request. Its about monitoring Kubernetes logs using EFK stack. Not sure if its going to be helpful but might give you some direction. You can check that out in the below video. kzbin.info/www/bejne/bp-6mWR9gclqpLs It sends all the logs to one index. I haven't tried configuring different indexes for different namespaces though. I think it is little bit involved. But the article in the below link attempts to do that by splitting logs for specific namespaces. See if that helps. blog.ptrk.io/tweaking-an-efk-stack-on-kubernetes/ Thanks.

[@josuastricker6023]

Do i need in the clients fluend bit vor logforwarding to fluentd?

[@theconfusedchannel6365]

Thanks good video. Do you have videos recommendation for enterprise architecture. How much space and how to distribute services if we are receiving 1GB of data/data etc..

[@jason11ac]

Hi, when I run docker-compose up -d, it shows all 3 being created successfully. But then when I run docker-compose ps, my fluentd container has the command "sh run.sh", shows state as Exit 1 and has no port information. Do you know why this is happening? I am running this on Red Hat Enterprise Linux. Don't know if this has something to do with it. Thank you in advance

[@justmeandopensource]

Hi Jason, thanks for watching this video. Elasticsearch requires the below kernel setting to be present before starting the container. I believe I mentioned this in the video. Can you make sure you did this step before starting the container? $ sudo sysctl -w vm.max_map_count=262144 Sometimes when I forget to run this command, elasticsearch container exits similart to your case. So I believe this could be a possible reason. Thanks.

[@andresmora6424]

Hi i follow the tutorial but in Kibana only exist the .kibana_1 i can see logstash index-name, the 3 containers are running

[@neppalivasanthalakshmi8407]

Hi , i am using fluentd office image and it is not recognising elasticsearch plugin showing error like unknown plugin elasticsearch. Does fluentd official container does not provide the elasticsearch plugin as default plugin?if not ,how to make my fluentd to recognise the plugin without rebuilding the image

[@venkatk8314]

Dear venkat, when i try to deploy fluentd using helm chart by default it deploys as root user. i changed to non root user by using security context. But the problem is it errors out "[error]: #0 unexpected error error_class=Errno::EACCES error="Permission denied @ rb_sysopen - /var/log/fluentd-containers.log.pos". Can you please advise in case if you had this issue

[@ulkaasati8509]

@Just me and Opensource : Hi Venkat, This is good video. Can you please guide how can I send application logs inside docker containers (not kubernetes cluster) to fluentd?

[@AlexLDemise]

Sorry for erasing the comment, but I had discovered the solution, and after that I saw an e-mail with yout comment. Thank you very much.

[@justmeandopensource]

No problem at all. Glad that you got the solution. Was it any different to the one I suggested? I thought it would be helpful for other viewers. Cheers.

[@AlexLDemise]

@@justmeandopensource Exactly the same, hehehe

[@justmeandopensource]

@@AlexLDemise Cool.

[@johnfinny100]

Hi, Im trying to install EFK stack docker container in AWS Vm and td agent in ubuntu Vm in aws. I want to know the system requirement for EFK stack if t2.micro is sufficient or medium is needed.If any other pre requisiste needed please let me know , Thanks !

[@justmeandopensource]

Hi John, thanks for watching. It depends on how much of data you are going to process in your elasticsearch. Just a bare minimum installation of elasticsearch itself requires half a gig of ram. t2.micro comes with 1GB RAM. So you will be fine installing just elasticsearch on it. But if you want to install other components as well, the machine could struggle.

[@prabhusamarla]

I ran docker-compose file and containers are up but unable to access kibana due to message I am doining this in my VM 2021-04-22T17:48:02Z","tags":["license","warning","xpack"],"pid":1,"message":"License information from the X-Pack plugin could not be obtained from Elasticsearch for the [data] cluster. Error: Request Timeout after 30000ms"}

[@MM-Mysore]

Thanks Venkat. It was very helpful. Would it be possible to explain xml parsing when the logs are present in multi lines. Currently, finding it very difficult to have the multiline xml logs are being parsed.

[@justmeandopensource]

Hi Manohar, thanks for watching this video. I haven't dealt with a multiline log file yet. But I guess you would use logstash to parse and format it before sending it to the elasticsearch engine. When I searched online, I found this stackoverflow discussion which might give you some direction. stackoverflow.com/questions/42645366/filebeat-process-multilne-xml Thanks.

[@MM-Mysore]

@@justmeandopensource Thanks Venkat. However, we are exploring with Fluentd plugins since we've chosen EFK stack. Currently, we are not able to get the XML parsed when its present in multiline. Tried using concat plugin and then we're unable to parse xml using available plugins.

[@justmeandopensource]

@@MM-Mysore Have you checked the below link on how to use multiline parser plugin for fluentd? docs.fluentd.org/parser/multiline I haven't tried it myself but might give you some idea. Thanks.

[@RS-jb7ph]

How fluentd communicates with kibana ? , and my requirement is i have only one log file , how can i place that logfile in fluentd without td-agent ?

[@udaymailforu]

Thanks Venkat for the detailed explanation. I have a question. Is FluentD log aggregator is mandatory to install? can i use only td-agent forwarder in all the nodes to send logs to elasticsearch?

[@justmeandopensource]

Hi Uday, thanks for watching. Yes using td-agent is abolutely fine.

[@udaymailforu]

@@justmeandopensource Thanks Venkat for your reply, your explanation is crisp and clear. It would be more helpful if u can talk about elk best practices in upcoming postings.

[@justmeandopensource]

@@udaymailforu Sure. At the moment, my focus has shifted to Kubernetes, GCP and AWS. I will do ad-hoc videos in this series whenever I get some time. Cheers.

[@JohnSmith-fv6td]

A big thumb's up on your video. I have a query, hope you can help. The current setup which I did use Elastic and kibana services from AWS, installed td-agent on app containers and make the logs flow to kibana with no issues. Do you think is this an effective solution without setting up log aggregator in between?

[@justmeandopensource]

Hi Naresh, thanks for watching this video. Fluentd can be used either as log forwarder or as log aggregator. Your set up is fine. You are using td-agent to forward the logs directly to elasticsearch. No problem with that. In the video, I forward logs to log aggregator which forwards it to elasticsearch. The use of aggregator in between will be useful in certain scenarios. An example would be high availability option. If you have 100s of clients forwarding through td-agent to elasticsearch and if you decide to change the elasticsearch url or port or any other aspect, you would have to update it on all 100s of clients. If you had used a log aggregator, you only need to update one machine or couple depending on HA setup. Hope this makes sense. Thanks, Venkat

[@JohnSmith-fv6td]

@@justmeandopensource Got it, thanks. Looking forward to more videos from your channel. Cheers!

[@nguyenhuy-pv7wc]

Hi Venkat, appreciate your video! However, when I try to replace the elasticsearch and kibana tag from 6.5.4 to version 7.0.0 and up in the docker-compose.yml file; the elasticsearch won't run and it exits with error code 78. Although I tried the "sudo sysctl -w vm.max_map_count=262144" but still the same error. Have you ever tried your guide with afterward elasticsearch version like 7 and later?

[@justmeandopensource]

Hi Nguyen, thanks for watching. I am sure I have tried v7. Have a look at my docker compose files in the below github repo. github.com/justmeandopensource/elk/tree/master/docker

[@nguyenhuy-pv7wc]

it works! tks Venkat

[@justmeandopensource]

Cool. No worries.

[@saiteju8169]

Can you please setup EFK in GKE k8s cluster in future videos?

[@kaycay014]

Venkat when i ran docker-compose up -d all the components are up except elasticsearch . i am able to see it exits with error code 78. when i checked for elasticsearch logs it is showing that bootstraps check failed. can you please help me on this ?

[@justmeandopensource]

Hi Kay, thanks for watching. I think you missed one important step without which elasticsearch container won't run properly. I am not sure whether I mentioned in this video. I have mentioned that on few of my elastisearch videos. Please make sure to run the below command on your host machine (assuming your host machine is Linux). $ sudo sysctl -w vm.max_map_count=262144 And then give it a try. Cheers.

[@Babbar_supreme_leader]

@@justmeandopensource Venkat you are awesome!

[@justmeandopensource]

@@Babbar_supreme_leader Thanks for watching.

[@sainathchandolu3096]

Great video. We follow the pattern to visualize logs Fluentd -> amazon kinesis -> logstash -> elastic search -> kibana. Do you have a video which has the above process? We use docker in an ec2 instance where our logs which is application logs are contained in two containers. Suggest me with the process Thanks.

[@justmeandopensource]

HI Sainath, thanks for watching this video. I haven't used Amazon kinesis. There is another video I did which is basically FileBeat -> Logstash -> Elasticsearch -> Kibanana. Thanks, Venkat

[@sainathchandolu3096]

Just me and Opensource thanks for the response! We recently switched from filebeat to fluentd

[@sainathchandolu3096]

I have a question, I have a docker running inside an ec2 instance. I pulled fluentd artifactory and created a container but I could not able to see td-agent installed. Are td-agent.conf and fluent.conf are two separate configuration files? Please provide me your valuable response? Thanks!

[@najamawan]

I am wondering how to setup password on kibana and elastic search so dont if using this on production data is not visible to public.

[@justmeandopensource]

Hi Najam, thanks for watching. I actually released a video today on how to deploy Elastic Cloud on Kubernetes. In this process, we are not using helm charts. We deploy Elastic operator which can manage our elasticsearch, kibana deployments in the cluster. This method of deploying elastic cluster comes with security and encryption by default. You can watch the first part of the video in the below link. kzbin.info/www/bejne/p5vRhWOmipV8iNE The second part where I will be explaining about filebeat configuration will be released next Monday. Cheers.

[@calljsaran]

Hi,,can you pls explain how to install and run fluentd on Mac machine?

[@justmeandopensource]

Hi Saravana, thanks for watching. There is this documentation that tells you how to do exactly that. docs.fluentd.org/installation/install-by-dmg Hopefully you will get it done. Cheers.

[@AlexLDemise]

Do you know where Could I get information to configure a haproxy server in Centos 7 to use td-agent and send logs to my docker/fluentd? In your video, you do the example with a rsyslog, but the machine that I have now to try it is a haproxy. thanks

[@justmeandopensource]

Hi Alex, I am kind of lost on what you are asking. In this video, I used td-agent on the client and it listens on a port. Any application on the client can forward the logs to this td-agent which inturn forwards it to the fluentd log aggregator and then to the elasticsearch. Haproxy is a load balancing solution. What is it you are trying to do with haproxy or where is it you are configuring? Thanks.

[@AlexLDemise]

@@justmeandopensource I wanna send the haproxy logs to my fluentd

[@justmeandopensource]

@@AlexLDemise Ahh now I got it. Let me see how that can be done and will let you know. Thanks.

[@AlexLDemise]

@@justmeandopensource Ohh, thank you very very much :)

[@justmeandopensource]

@@AlexLDemise You can follow two approaches to get haproxy logs to elasticsearch. 1. Follow the article www.fluentd.org/guides/recipes/haproxy-elasticsearch. Sending logs to td-agent directly 2. Configure haproxy to write logs through rsyslog and configure rsyslog to forward to td-agent. You can check this link www.haproxy.com/blog/introduction-to-haproxy-logging/ Thanks.

[@najamawan]

hey, my host doesn't have fluentd installed but since i follow your video i have it running as log aggregator. what i want is to send logs from containers (launching on host) to aggregator. following command i tried and got error. is this possible? $ docker run hello-world --log-driver=fluentd --log-opt fluentd-address=192.168.0.42:24224 docker: Error response from daemon: OCI runtime create failed: container_linux.go:346: starting container process caused "exec: \"--log-driver=fluentd\": executable file not found in $PATH": unknown. ERRO[0012] error waiting for container: context canceled

[@najamawan]

docker run --log-driver=fluentd --log-opt fluentd-address=192.168.0.42:24224 hello-world its working and i can see output in kibana. :)

[@justmeandopensource]

@@najamawan Cool.

[@blockadsshop]

Thank you for sharing this video, Im beginner for EFK stack and this video is easy to understand. which are you recommend to use logstash or fluentd for collector service? Im consider about which one I should learn first. Hmmm About performance, usage CPU and Memory

[@justmeandopensource]

Hi, thanks for watching. I would advice you to try each of those you mentioned and decide yourself which suits your needs. There is no one best solution and I do not know full context of your environment.

[@blockadsshop]

@@justmeandopensource how about if my context is not too much CPU and Memory.

[@blockadsshop]

Anyway, I will try EFK first, just follow your video. Thank you so much.......

[@leegaines2391]

how does the td-agent know the ip address and port of the fluentd server? i didn't see that information in td-agent config...

[@justmeandopensource]

Hi Lee, The ip address of the fluentd server is configured in the td-agents config file clients-td-agent.conf. In the section under section, you specify the ip address of the fluentd server. Starting at 20:00 you can see me configuring td-agent. Cheers.

[@leegaines2391]

@@justmeandopensource ah I see now. But no Port was specified... I assume the port is conventional and td agent knows what port to default to..?

[@justmeandopensource]

@@leegaines2391 yes. Thats right.

[@AshokKumar-ez5yu]

how to set log rotation period on fluentd

[@justmeandopensource]

Hi Ashok, thanks for watching. I have gone with the default setting where log rotation is not enabled. The below documentation link might help you in right direction. docs.fluentd.org/deployment/logging#log-rotation-setting Cheers.

[@praveenkorvi2227]

Hi Venkat, I have a kubernetes setup configured with EFK by using helm charts to monitor application logs. But one of the application is hosted on outside of cluster on a VM with windows OS. Is it possible to forward the logs of windows VM to EFK which is configured in Kubernetes cluster

[@justmeandopensource]

HI Praveen, thanks for watching. Thats absolutely possible. You just need to make sure to expose your elastic search endpoint outside the cluster.

[@praveenkorvi2227]

@@justmeandopensource Hi Venkat, thanks for your response. Which one would be better? filebeat, td-agent or winlogbeat? Which service should use to expose elastic search (nodeport or load balancer)?

[@justmeandopensource]

@@praveenkorvi2227 Exposing a service through load balancer is the standard practise. What is it exactly you want to collect (metrics/logs or anything else). Each tool is designed for a specific purpose. Cheers.

[@praveenkorvi2227]

@@justmeandopensource .NET Application logs from Windows server. Thats it

[@justmeandopensource]

@@praveenkorvi2227 Don't have much experience with .NET but i would guess winlogbeat would be better in your case. Cheers.

[@nileshyadav9764]

launch images:centos/7 efkclient this command is not running mac book what to do??

[@justmeandopensource]

Hi Nilesh, thanks for watching. I used LXC containers to launch a machine. You can use a CentOS7 Virtual Machines for this.

[@nileshyadav9764]

@@justmeandopensource can u suggest give me cmd for that

[@justmeandopensource]

@@nileshyadav9764 You just need to launch a virtual machine using VirtualBox.

[@shafigh6916]

I can not access Kibana dashboard and error i get is: Kibana server is not ready yet... any idea?

[@justmeandopensource]

Hi Sha, thanks for watching. Please check the logs of Kibana container which might give you some clue about the failure.

[@shafigh6916]

@@justmeandopensource if i knew what is going on in the logs, definitely i would have not taken your time

[@justmeandopensource]

@@shafigh6916 It would be helpful if you could share the elasticsearch and kibana logs

[@HosseinOjvar]

Hi Is it possible to you to create a tutorial about to make a desktop like your own? Your desktop is so cool ;)

[@justmeandopensource]

Hi Amir, I keep switching OS and desktop environment. I will never be satisfied and contended with one. And this one was my setup a very long time ago and not using this at the moment. I am currently on Archlinux running I3 tiling window manager :)

[@surisurendrababu]

how did you added those widgets in your screen could you show us please

[@justmeandopensource]

Hi Surendra, thanks for watching. Its the conky process. Unfortunately I don't have the conky config file anymore. You should be able to get a basic conkyrc file in the internet and customize it to your needs.

[@surisurendrababu]

@@justmeandopensource ok thanks and please make videos on docker+ansible related and elastalert+elk give a try !!

[@justmeandopensource]

Sure will do.

[@surisurendrababu]

@@justmeandopensource Thanks will be waiting for those !!

[@muthukumarramu3149]

Superb Video.. I need to gather logs from application container... in that how can i install the td agent inside the container.. the application containers are in production.. Kindly help me

[@justmeandopensource]

Hi, you don't want to install td agent inside all the containers. Instead you can run it as a daemonset in your cluster that will pull container logs from all the pods. I am planning to do a video on that at some point. Cheers.

[@muthukumarramu3149]

@@justmeandopensource thanks bro.. please put video for docker containers also... my environment running on docker..

[@justmeandopensource]

no worries.

[@ulkaasati8509]

@@muthukumarramu3149 How did you do it? I too have docker container env and want to send application logs inside docker container to fluentd. Pl guide

[@muthukumarramu3149]

@@ulkaasati8509 two ways can be done.. One is graylog and other one EFK. In EFK all container network should be same. Then only you can get the logs.. for application logs.. developers needs to write the code and logs stored in volumes.

[@chillyvanilly6352]

But what u didn't mention regarding Fluentd, is that the config-part of `logstash_format true` is important! Otherwise Kibana won't be able to fetch it from Elasticsearch!

[@justmeandopensource]

Hi, thanks for watching and pointing out this info. Will be helpful to others as well. Cheers.

[@chillyvanilly6352]

@@justmeandopensource Thank you for the video my good Sir! Glad I could help :)

[@thunderbirds8633]

can you create a video on azure insights also

[@justmeandopensource]

Hi, thanks for watching. I am not into Azure yet.

[@surisurendrababu]

Please Make same Video in Ubuntu i have 40 employees in my office i need to set all those with beats

[@justmeandopensource]

Thanks for watching. I will add it to my list. Cheers.

[@surisurendrababu]

@@justmeandopensource when can we expect 😉😉

[@justmeandopensource]

@@surisurendrababu Can you explain what exactly you want?

[@surisurendrababu]

@@justmeandopensource I want to install Metric and audit beat in my employees laptop and through that i want to check the logs

[@najamawan]

➜ docker-compose up -d Building fluentd Step 1/2 : FROM fluent/fluentd ---> 9406ff63f205 Step 2/2 : RUN ["gem", "install", "fluent-plugin-elasticsearch", "--no-rdoc", "--no-ri"] ---> Running in 2eeaa6316492 ERROR: Could not find a valid gem 'fluent-plugin-elasticsearch' (>= 0), here is why: Unable to download data from rubygems.org/ - no such name (rubygems.org/specs.4.8.gz) ERROR: Service 'fluentd' failed to build: The command 'gem install fluent-plugin-elasticsearch --no-rdoc --no-ri' returned a non-zero code: 2 first command failing hard.

[@justmeandopensource]

Hi Najam, thanks for watching. It might be a transient issue. I checked the url rubygems.org/specs.4.8.gz and I could download the tar file. So it could just be a temporary issue when you tried. Please try it again. Cheers.

[@najamawan]

@@justmeandopensource yeah i can open in browser and in cli using wget. but when using docker compose up. it throws error. can you try docker compose up and see .

[@justmeandopensource]

@@najamawan Its working fine for me. pastebin.com/h5fpfDUn

[@najamawan]

@@justmeandopensource i m using opensuse tumbleweed i disable firewall restarted issued docker-compose up and everything works. cant believe wasted 2 days for this. thanks for your response and your test. i was not hoping you would reply but you did and thats awesome :)

[@najamawan]

@@justmeandopensource elasticsearch | [1]: max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]