Setup Filebeat to Monitor Elasticsearch Logs

[@MrSalFav]

Hi Ali, Thank you for putting out these videos they are really helpful . wanted to learn more about Elastic Stack for my ForgeRock project. and you videos are of great help.

[@AliYounesGo4IT]

I'm very glad brother Saleem you found them helpful. I wish you the best with your project!

[@arunrmyt]

Hi Great video again

[@kumak9890]

Hi Ali, Thank you for uploading videos about ELK.. Hopefully you can upload a monitor log with Elastic Agent.

[@onemo]

Hey, excellent video, the entire ElasticSearch saga is really helpful! For Metricbeat, you skipped part 2.2 (setup.kibana) and part 3, was it intentional or you realized afterwards that you should have done the same?

[@AliYounesGo4IT]

Thanks, I'm glad you found them helpful! That step is optional, it allows loading of pre-built dashboards in Kibana, you'll need to setup a user with the kibana_admin built-in role or equivalent privileges on the cluster.

[@mnm27mnmpth]

Thank you so much for explaining filebeat. Can you please put a video tutorial, how to connect filebeat to API GATEWAY?

[@szymonzalewski9374]

Hi Ali, does it need to be set up on other nodes aswell?

[@ananyayechuri320]

Hey, I was able to download and setup filebeat and it showed me that kibana dashboard must be running and reachable but when I refresh the page the logs section shows me that I still need to install filebeat which I have already done

[@walidbarrani7769]

hey ali thanks for the video i just wanna ask before i start the steps did it worked when i just want to see logs with suricata on another filebeat machine (ubuntu i used ) your answer will be helpful thank you

[@AliYounesGo4IT]

Hello Walid, it should work as long as you have a connection to that destination machine, and send the logs to the right port (with firewall allowing data to that port)

[@tonylagumen2810]

Hello Ali - you are setting up filebeat on chamber2 but the monitoring log dashboard in kibana is showing chamber1. --- Was the filebeat log shipper configuration properly setup for chamber2?

[@AliYounesGo4IT]

Great question, the filebeat module monitors the logs locally on the node and sends those logs to an index on the cluster. The cluster can allocate the shards of that index to any node, I believe that's why the logs showed up on Chamber1.

[@SnakeFredy]

Thank you very much for the videos Ali. I wanted to ask you, I have mounted Elasticsearch, kibana and logstash, Is it better to replace logstash by Filebeat? this since I also have a Fortinet Firewall. Thanks in advance

[@AliYounesGo4IT]

You're welcome John! I like filebeat better than logstash, because it has a module for Fortinet, it parses the fields, and adds information about Geo locations or IP's. Logstash is for specific situations where you need to add more inputs and enrich your logs from another systems

[@SnakeFredy]

@@AliYounesGo4IT Thank's a lot, Ali!

[@unlimited.travel.channel]

I can't get "admin login alerts" with Filebeat. Which Fortigate syslog parameters should be enabled? Thanks in advance.

[@MyTeevo]

Hey appreciate your efforts, your videos are extremely informative. Could you please do a detailed video on setting up interface stats for fortigate on ELK

[@AliYounesGo4IT]

I can work on that. What stats are you interested in seeing?

[@MyTeevo]

@@AliYounesGo4IT List interfaces, Interface up or down... Bandwidth in - out.... Cpu/memory and other health checks... Am working on a project with the exact same scenario... Will be able to help me...

[@AliYounesGo4IT]

@@MyTeevo I'm planning on testing out SNMP with Logstash, so I will try that with my fortigate and made a video

[@MyTeevo]

@@AliYounesGo4IT sounds great... Eagerly waiting... And we could setup input from syslog and snmp on the same dashboard?

[@MyTeevo]

@@AliYounesGo4IT Hi Ali... Is there anyway i could talk to you...

[@romanjkee6211]

Great video! Need it with logstash) Also, how can I monitor apm queue free size?

[@nisrrah8198]

Where are the imported logs from fluentd or filebeat stored? In logstash or elasticsearch? I need to configure this so that it doesnt fill up the c:

[@clearthinking5441]

why don't you configure all of this through docker?

[@AliYounesGo4IT]

I'm still testing with Docker, I will make a video soon!

[@arunrmyt]

Can you help us with fortigate logs are not working with var.input: file and var.path: /path/to/*.log

[@JoseManuel-lo2ed]

Do not work to me Mister, ELK is horrible. I do not what to do more to make that filebeat send logs to my elasticsearch... Amazing. Best regards.

[@JoseManuel-lo2ed]

You are the ElasticBoss... Jajajajajajajaja. best regards.

[@zmartinelli]

I think you forget to give root permissions on the directories, that is why the elasticsearch data did not show in the UI.

[@AliYounesGo4IT]

They showed up at the end of the video! Maybe it needs some time to refresh the view

[@IvarsRuza]

Nice vidtuts! But why U R enabling the same repo again and again? Alos for ELK REPP - make sure: enabled=0, so U will not update it automaticly. When U will be ready to update ELK, use: yum --enablerep=Eelastic-8.x install filebeat, metricbeat logsthas elasticsearch etc.

[@AliYounesGo4IT]

Thanks for the tip! I'm not very advanced in Linux :)

[@kumak9890]

Hi Ali, Thank you for uploading videos about ELK.. Hopefully you can upload a monitor log with Elastic Agent.

[@AliYounesGo4IT]

No problem, I will work on it soon