Enrich your Data in Elasticsearch

#elasticsearch #filebeat #kibana #elasticsearchtutorial #logstash #metricbeat
In this tutorial, we setup data enriching in Elasticsearch to have more insights and richer data sets that will provide more understanding and relevance.
Watch how to visualize FortiGate logs on Kibana:
• Visualizing FortiGate ...
Thank you for watching!
Follow my Twitter: / ayounes9
Follow my LinkedIn: / aliyounes9

[@systechadmin8368]

Thanks !! another beatifully explained Al Sir videos .

[@oolyo6604]

Thanks Ali for this great elasticsearch tutorial, I would love to see you making video about datastreams and index template.

[@mhenimerzouki1285]

Great video i love these ELK series, i would love to see you cover compression with Elasticsearch lots of logs that needs to be stored for a long time i'm exploring the index.codec parameter and was wondering if it's possible to apply different compression algorithms depending on the ILM phase the index is in (less compression but fast access in the hot phase, and the better compression in the warm and cold phases).

[@samsal073]

Hi Ali, Do you think ElasticSearch can be used for documents archiving and records management. I work for engineering company where project can produce 10s of thousands of documents . All those need to be archived provided retention schedule before records\files are destroyed (deleted from they system with log documenting the destruction event).

[@foxwhite25]

Thanks for the guide, btw just to add to this, if you wants your index to automatically use the pipeline by default change the index.default_pipeline setting

[@Ethan777100]

Hi Ali, Could I use the Enrich function to do Data Cleaning? For example, I already know my incoming logs may have rows of false positive entries that do not reflect accurate state of system. Hence I want to have a Policy to delete such rows based on correlation with rows above/below such that only clean, accurate data is ingested into Elasticsearch.

[@AliYounesGo4IT]

Hi Ethan, I believe the enrich processor can add data to an index, not clean it. There are other processors to remove fields or if you're using Logstash you can drop fields when they have a specific value with if statements. I hope that helps.

[@Ethan777100]

@@AliYounesGo4IT Didnt know LS can do this. U got video to demo this feature? Any links to official documentation?

[@mahdiar3740]

Please say about send suricata logs as a nids to ELK . THANKS

[@lucaskayser]

ERROR: Failed to determine the health of the cluster.???????????????????