Terraform 11 - Deploy ELK (ElasticSearch, Logstash & Kibana) stack on AWS via Terraform

This video we went through how do we deploy ELK stack to AWS manually. Here is the key steps to review:
1. sudo apt-get update && sudo apt-get upgrade -y
2. sudo apt-get install default-jre -y
3. Install ElasticSearch, Logstash, MetricBeat and Kibana followed the instruction on documentation via package manager:
www.elastic.co/guide/en/elast...
4. Start each ELK stack as service.
Example of logstash config:
/etc/logstash/conf.d/apache.conf
input {
file {
path = "/home/ubuntu/apache-daily-access.log"
start_position = "beginning"
sincedb_path = "/dev/null"
}
}
filter {
grok {
match = { "message" = "%{COMBINEDAPACHELOG}" }
}
date {
match = [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
}
geoip {
source = "clientip"
}
}
output {
elasticsearch {
hosts = ["localhost:9200"]
}
}
Timestamp:
0:00​ - Intro
05:42 - Spin up EC2 instance via Terraform module
14:41 - Install & Configure ELK stack on EC2 instance
28:46 - Configure index pattern on Kibana

[@jordanbooker3195]

I am finding trouble with getting the ubuntu server to spin up. It says there is something wrong source code that was copied from github.

[@tkhalane]

hi, what do you suggest as the best of managing the Logstash pipeline configuration, i.e being able to update it and apply it to a running ec2 instance

[@zztalk]

I am using Terraform + Ansible + Parker.

[@tkhalane]

@@zztalk thank you. Do you a github repo for this?

[@zztalk]

@@tkhalane I shared the key config files in the description of this video. Check it out to see if you get what you're looking for..

[@Missing-link]

Can you please share the git repo for the modules?

[@dataadvocate2764]

Do you have any such video for On premises ELK cluster using terraform?

[@zztalk]

You needed to figure out if terraform has a provider to support your "On Premises" env.

[@jordanbooker3195]

Also, do you mind sharing the variable part of your code. I want to see if I am missing something.

[@zztalk]

cat variable.tf variable "AWS_REGION" { default = "us-east-2" } variable "PATH_TO_PUBLIC_KEY" { description = "Public key path" default = "mykey.pub" } variable "PATH_TO_PRIVATE_KEY" { default = "mykey" } variable "AWS_ACCESS_KEY" { type = string default = "xxxx" } variable "AWS_SECRET_KEY" { type = string default = "xxxx" } variable "INSTANCE_USERNAME" { default = "ubuntu" }

[@jordanbooker4711]

@@zztalk Thank you so much!

[@user-yi7jf5ht9k]

Installing new version of config fiel

[@cloudfhytech2520]

Please provide git link

[@jordanbooker4711]

Also when I do terraform apply I get this error: Error: Error launching source instance: InvalidAMIID.NotFound: The image id '[ami-0e472ba40eb589f49]' does not exist │ status code: 400, request id: fcb2bcfd-25ef-444e-be99-fe93ee52d94a │ │ with module.ec2_instance.aws_instance.this[0], │ on .terraform/modules/ec2_instance/main.tf line 5, in resource "aws_instance" "this": │ 5: resource "aws_instance" "this" { I dont know why I get this error I changed the subnet to my subnet and I copied the ami id. I think there must be something wrong in the Github code.

[@zztalk]

The error msg complains about the AMI. Were you spinning up your EC2 in the same availability zone and same region? Double check and get a valid AMI that is available on your AWS console. That should fix the issue.

[@user-yi7jf5ht9k]

Installing new version of config file /etc/kibana/kibana.yml ... Restarting kibana service... OK Created Kibana keystore in /etc/kibana/kibana.keystore Scanning processes... Scanning linux images... Running kernel seems to be up-to-date. No services need to be restarted. No containers need to be restarted. No user sessions are running outdated binaries. No VM guests are running outdated hypervisor (qemu) binaries on this host.

[@user-yi7jf5ht9k]

not able to install kibana..getting this error.how to solve it