Logstash: Path to ECS for 8.0

The Elastic Common Schema is a community-driven effort to provide consistent semantic meaning to datasets so that data from disparate sources can be meaningfully used together. In Logstash 8.0, ECS compatibility is on-by-default - this is a pretty major change to how many plugins operate.
In this talk, we outline the rationale behind the transition and also highlight how to opt-OUT of the transition with a simple pipeline setting.
Reference links:
- Demo scripts: github.com/yaauie/p2ecs-demo

[@wylde780]

I really appreciate the effort put into keeping existing implementations functional while still moving ahead and innovativating.

[@peterjakubik]

Thanks for great explanation

[@whatthef911]

This project is great for accepting configurable inputs. I work with legacy code which cannot integrate with APM and I am able to store and analyze a variety of steaming log files.

[@logeshlogan9824]

hi "set": { "field": "field1", "value": "{{ endpoint | regex_replace('/live/disk1/[^/]+/(?[^/]+)/', '$output') }}" } .im try to get the value in field1 but getting empty reply and also if remove the regex its working fine .can u help em

[@trixiemp890]

How about managing logstash pipelines into multiple logstash nodes? We have two logstash nodes and everytime we update a pipeline in logstash server 1, we need to manually update the logstash pipeline in server 2. It’s very time consuming

[@ductapesuprhero]

There are a couple paths forward to simplify: One is to use Kibana's Central Management features, and to configure Logstash to "subscribe" to pipeline definitions by name -- when the config changes in Kibana, both instances will notice the changes and restart their pipelines with the new definitions. The other path is to use a source-control management tool like "git" to "commit" changes once, then to "pull" your changes down on teach Logstash host (manually, or by a cron-scheduled script); with `config.reload: automatic: true`, Logstash will similarly notice the changes to files on disk and reload the relevant pipeline. For more details, I'd recommend asking in the community forums -> discuss.elastic.co/c/elastic-stack/logstash

[@SoCalCycling]

Ye until now it has been much easier to just dev code rather get into the rats nest of logstash pipelines and plugins... has always been horrible

[@namelastname4077]

I effin hate logstash. The whole Elastic stack SUCKS