Using Index Lifecycle Management (ILM) with Logstash

#elasticsearch #elasticsearchtutorial #logstash #kibana #filebeat
This video shows how to setup ILM with Logstash configuration files.
Watch how to install and setup Logstash to ingest Fortinet Syslogs:
• Installing and Configu...
Watch how to setup Index Lifecycle Management ILM in Elasticsearch:
• Setting Up Elasticsear...
Thank you for watching!
Follow my Twitter: / ayounes9
Follow my Blog: www.thelionping.com/

[@ahmed_mansour5]

Thanks a lot for the great explanation! It was really useful as it gave the two ways to deploy ILM (with and without rollover)

[@tyserie9057]

Love your work. Please keep them coming.

[@Smaug1248]

Nice video Clear sound. Thanks

[@chikugerson5291]

Great...you got yourself a subscriber 👏👏

[@dipdiptortillachip]

Great vids!

[@miguelsaiz8151]

Great Videos !

[@user-ul4uv6xi7e]

Great!

[@AdolfoCuadra-tw7cn]

Nice video! I am trying to create a rollover every week. I have created the template and policy just as you have. In the logstash.conf file, if I set ilm_pattern => “{now/day}-000001” just as you have , and the ilm policy max age to be 7 will the index rollover name be the date it is rolled over with the trailing 000001 or will the trailing numbers just keep incrementing when it rolls over and the date stays the same?

[@rahulsirugudi]

I am using input beats, i have done exactly all the steps but the problem is after i start logstash i don't nothing coming to the index but i can see doc count and size are increasing but from discover i don't see any. Also it broke other indexes not sure what i missed.

[@veyselyuksel1198]

Hi, Thanks for sharing. But we use rollover mechanism for shard optimization and performance tuning. You said that logstash creates index and you don't like pattern numbers :) But firewall-2022.11.08 index has static shard number. There are many shard limitations in elasticsearch. If you dont use them, your system is getting slower day by day.

[@mighnmagic9430]

Can we remove from indice settings and template rolloover alias if we dont use rollover in firewall policy ?

[@user-mt2gc5em6v]

Hi , I have a query If we are using Index with date and we need to apply Rollover in hot phase on that index. is it possible to apply or we can apply only delete phase ?

[@SalmanIsha]

index => firewall [ which you set as a rollover alias ] will write the data to 0001 and subsequent indexes that will be active on that particular moment.

[@mighnmagic9430]

Hi, if in Index Template i have multiple index patterns, for example test*, number* and word* and I have same data views and I want to use same policy , do I update each config with the same ILM policy ? And I do for each: PUT test-000001 { "aliases": { "mypolicy": { "is_write_index": true } } } PUT number-000001 { "aliases": { "mypolicy": { "is_write_index": true } } } ?

[@abdel8063]

Thanks (y)

[@IvarsRuza]

Nice vid! That Rollover is for data streams and be sure U set Your alias. Based on Your alias build dashboards, search quires, visualizations and never brake. With alias You target all Your indices with the same name. Data streams are great for managing, but for me somehow they are slow on warm on cold nodes. Data streams are like Timescale for Postgresql

[@mighnmagic9430]

Hi, if in Index Template i have multiple index patterns, for example test*, number* and word* and I have same data views and I want to use same policy , do I update each config with the same ILM policy ? And I do for each: PUT test-000001 { "aliases": { "mypolicy": { "is_write_index": true } } } PUT number-000001 { "aliases": { "mypolicy": { "is_write_index": true } } } ?

[@praveenkumar-uc3tu]

I have a clarification, the same way we did for index gets created everyday also have created only life cycle with deletion of 30 days. Also have assigned same ILM assigned to index but it’s not automatically managed by index when the new index gets created. Please advise

[@clearthinking5441]

what are your personal views on self-managed ELK stack vs payed service?

[@fabmartel]

thanks, very good explain. But plz reduce the size of webcam circle plzzzzz...and put right top... we dont see your windows console

[@AliYounesGo4IT]

Thank you for your suggestion, i will do that!

[@pokem9947]

I have ELK stack where logstash is 6.x version and kibana is 7.17 version The index i have mentioned in logstash conf file is not visible in kibana index pattern. How to fix?? Please help. I am new to this

[@akshaysaini1613]

Hi, please reply me I have trying so many time in ilm policy.. my elk version is 7.5.1 my indicis will move hot to warm. But in warm to cold it's not moving.even i have set min_age only 2m for translation warm to cold .....i have stuck this issue before a months please resolve it...and video on that like your indica will transfer all indicis

[@hasanidriss1519]

nice job man

[@AliYounesGo4IT]

Thank you!

[@hasanidriss1519]

@@AliYounesGo4IT hi ali plz if u can make a video of rules and alerts, also for email alerts what is the best practice for the free license, and could we integrate it with with an open source ticketing system ?

[@fabmartel]

I have a more complex question, I would like my index pattern not to be today's date, but the date that is in the message line. Because if Logstash does not know how to access the remote elasticsearch if I restart logstash the next day, the lines stored in the logstash queue or disk queue will be indexed the same day, except the index must be based on the true log date in the message .

[@AliYounesGo4IT]

That's a good question! I think this needs some if statements and variables inside the logstash config file, and send logs to the appropriate index if they match that date. I never encountered this before but I will try it in the lab and see how it goes.

[@splendx]

Hello friend! It would be great if you showed how to send sql database via logstash and filebeat converting them to json. And how to make a full-text search on the site

[@AliYounesGo4IT]

I will work on that!

[@omeroncel8569]

Hi , Do you plan to make a video about apm with elasticsearch? and fleet server. thank you

[@AliYounesGo4IT]

I will be covering Fleet and elastic agents soon, APM will be later

[@user-cr8mn8mi5o]

Can video "Using Index Lifecycle Management (ILM) with Elasticsearch"+Data Stream? pleas

[@markmarais8524]

Is it possible to make a video securing logstash with your elastic cluster(output) as well as securing communication between winlogbeat and the logstash(Input)

[@AliYounesGo4IT]

I will try to do that in the future

[@markmarais8524]

@@AliYounesGo4IT Thanks not alot of content on the secure communication between logstash and beats.

[@wylde780]

Hrmm just noticed the timezone value in your logstash conf. You are in Edmonton?

[@AliYounesGo4IT]

Yes, mountain standard time