Elasticsearch 8.x: Fleet Server + Logstash Output

[@evermightsystems]

Written Summary Here: elasticsearch.evermight.com/fleet-server-and-logstash

[@kamande_john]

I somehow managed to set Logstash output as the default output for Intergrations hence I don't need a license to use it!

[@sergeyb.4812]

Hi. For the given scenario - if I understand you correctly - you say that the Linux/Windows agents send the log data to the fleet server, and then fleet server sends the log data to Logstash. However, I believe it is not how it works. In fleet server you configure the output setting for agents, not for the fleet server itself. This configuration will be then used in the policies and will be brought to the Linux/Windows agents so that they get THEIR output centrally configured to use Logstash. In the Fleet settings the field "Outputs" also has a description "Specify where AGENTS will send data".

[@evermightsystems]

You might actually be right! I will review everything again. And if I'm in error, I'll unlist those video and post an updated version

[@KAINARTZ]

Is this method better for large environments? For example in my environment i have 2 agents with 20 integrations each recieving data from firewalls and another 300 agents that are on endpoints and servers collecting syslogs and metrics that send data to elasticsearch, however this does put a lot of load onto elasticsearch ingest nodes since they have to parse everything. Would introducing logstash push all the parsing over to logstash and reduce the load on elasticsearch nodes? Also when doing so will logstash use the ingest pipelines that come with integrations or does it just become a log forwarder and elasticsearch is still the one doing the parsing.

[@evermightsystems]

I don't think you need logstash in that case. The benefit of this logstash set up is if you wish to do some data transformations before sending it to elasticsearch, or if you want logstash to ship data to an additional location besides elastic search. Eg., from logstash you want to send one copy of data to elasticsearch and another copy To a MySQL database.

[@khaiphan8659]

Thanks for that. I have a question: can I install fleet server on the same host where elasticsearch and kibana is installed?

[@evermightsystems]

Yup we've kept fleet+kibana+elastic on the same server in some projects in our past. We even used the same FQDN/Hostname to reference all 3 services but just using different port numbers. Which means we'd only need one TLS certificate for that one FQDN/Hostname. Just make sure your server is powerful enough to host all those services simultaneously.

[@khaiphan8659]

@@evermightsystems when I install fleet server using ./elastic-agent install … command, I get the error “fleet-server failed: context canceled”. T saw troubleshooting guide but I couldn’t find my problem. Help

[@WSBRGamer]

does the agent send it to the fleet server > send it to logstash > logstash send it to elasticsearch?

[@evermightsystems]

I will post a correction when I get a moment. Another person on this thread corrected me and said the fleet server only sends an agent policy too the elastic agent. The policy tells the elastic agent to send data to logstash. Once the data is in logstash, then you can have logstash send the data anywhere else. I believe the actual steps I demonstrated in the video is correct (eg. what to click and what to type) , I just made a mistake describing the high level workflow. Once I upload an corrected video, I would unlist this video.

[@WSBRGamer]

@@evermightsystems Can I ask a question? Following situation: Remote Elastic Agent over the internet > logstash The input beats/elastic agent inside logstash will the port always be 5044? Because it seems to me that port 5044 cannot be changed. The Elastic agent sends data via https 443 over the internet, do you have to do port redirection from 443 to 5044 when it arrives in logstash?

[@ibnudafa8772]

request grok for fortinet

[@evermightsystems]

OK added to task list, but might be a while before we can get to it!