How to deploy a multi-node elastic-search cluster | elasticsearch deep dive | elk stack tutorial

#DevOps #Lifelonglearning #learningisgood #SecDevOps #CyberSecurity #LogAnalytics #Elasticsearch #elk #elk stack
Hey Guys
In this lecture elasticsearch deep dive, I will demonstrate How to deploy a multi-node elastic-search cluster on a Google cloud platform(GCP) compute engine VM using Debian Package. We will deploy a 3 node elasticsearch cluster with each elasticsearch node communicating over SSL and kibana communicating to our 3 node elasticsearch cluster over https.
00:00 Introduction
00:25 High level Steps for installation
01:12 Creating our virtual machines for this Demo
02:12 Testing connectivity to our GCP Virtual machines
03:17 Installation reference for our elk multi node setup
03:56 elasticsearch base configration
05:44 Generating CA for our elasticsearch cluster
06:26 Generating SSL certificates for our 3 node elasticsearch cluster
07:25 Generating HTTPS certificate for our multi node elasticsearch cluster
10:50 Copy the certificates on first node
12:45 Configure elasticsearch node 1 with security settings
14:47 Distribute certificates and elasticsearch.yml to our cluster nodes
17:46 Personalise elasticsearch.yml for each cluster node
19:10 Distribute certificates and configure kibana instance
21:13 start elasticsearch
21:40 Troubleshoot issues in your cluster configuration
24:06 Generate credentials for elasticsearch built-in users
24:34 Configure password authentication for elasticsearch on kibana
25:12 Test connectivity to our multi node elasticsearch cluster
This video is part of a playlist where I will demonstrate how to automate the deployment of ELK Stack (Elasticsearch, Logstash and Kibana ) and Beats( Heartbeat, Metric Beat, Packet beat, File Beat etc) using Ansible
Link to Playlist:
• Elasticsearch Automati...
Link to the file used in this video
gitlab.com/LabIT/elasticsearc...
Link to git repository
gitlab.com/LabIT/elasticsearc...
We will use Google cloud platform to host our VMs. GCP provides $300 in free trial credits which we will use to host our lab environment.
Elasticsearch
Elasticsearch is a distributed, RESTful search and analytics engine capable of addressing a growing number of use cases. As the heart of the Elastic Stack, it centrally stores your data for lightning fast search, fine‑tuned relevancy, and powerful analytics that scale with ease.
Ansible
Ansible is an IT automation tool. It can configure systems, deploy software, and orchestrate more advanced IT tasks such as continuous deployments or zero downtime rolling updates.
Please Like my video and subscribe to my channel and leave your comments.

[@LabITtraining]

To join discussion on elasticsearch please join join.slack.com/t/labit-training/shared_invite/zt-11lup6ihz-qSEDwuYlgSLBxkbWxHhkkQ If you like this course and would like to support me you can - Purchase my longer 5 hour course on Udemy : www.udemy.com/course/elasticsearch-101/?referralCode=2F96AD63FB30A07EDCEE - If you have a Skilshare account , you can access my course with the following link : Skillshare : www.skillshare.com/classes/ElasticSearch-101-A-Beginners-guide-to-elastic-stack/562700361/projects

[@inuroxas1]

I never comment videos, but I think you deserve some recognition. You explained it very clear and easy to follow. Thank you for your effort :)

[@carnealse]

Is there an update to show 8.x?

[@abhisheksinha3119]

Hey I wanted to know like can I monitor multiple es clusters using Kibana(basic version) or it's included in paid version?

[@sandeepbharmoria]

I want to set a elasticsearch cluster in docker swarm, nginx, 3 master, 2 coordination node, 4 data nodes, can u guide me sir?

[@wens8854]

Nice video thanks. You missed input one 9200 in conf file

[@supriyopaul1893]

This video was very helpful. Have been struggling to spin a multinode cluster for a week now. Can you please also inform me on how to generate the ssl certificates for connecting beats and publish data to the cluster?

[@LabITtraining]

If using self managed , generate it from the elasticsearch utility otherwise from your CA.

[@supriyopaul1893]

@@LabITtraining commands?

[@shobhithraj2162]

Hi I have gone through "How to deploy a multi-node elastic-search cluster | elasticsearch deep dive | elk stack tutorial" videos its very helpfull. how to connect logstash with Elastic search. Is there any video for same.

[@LabITtraining]

Hi Shobhith, Logstash videos are part of udemy course. Here is discount coupon if you like www.udemy.com/course/elasticsearch-101/?couponCode=7FFD043D07D3621B0266 If you dont want to purchase the course, you can look at the elasticsearch output for logstash, all you have to do is provide elasticserver URL and credentials to the output.

[@clearthinking5441]

Is it possible to achieve this setup through docker-compose? At the moment I can create my own multi-node cluster but the nodes are split into the same machine which fails to provide the benefits of horizontal scaling.

[@LabITtraining]

You can , there is a docket image available for elasticsearch

[@MrFathimohamed]

Thank you! Those who get any error cannot read configured [PKCS1 because of password like, you will need to reset all the keys without a password with the following commands: ./bin/elasticsearch-keystore add xpack.security.http.ssl.keystore.secure_password ./bin/elasticsearch-keystore add xpack.security.http.ssl.truststore.secure_password ./bin/elasticsearch-keystore add xpack.security.transport.ssl.keystore.secure_password ./bin/elasticsearch-keystore add xpack.security.transport.ssl.truststore.secure_password

[@4keka]

Is the cluster coming up properly? I am seeing below kind of errors after resolving certificate issues [WARN ][o.e.c.c.ClusterFormationFailureHelper] [elk-2] master not discovered yet, this node has not previously joined a bootstrapped cluster, and this node must discover master-eligible nodes [elk-1, elk-2] to bootstrap a cluster: have discovered [{elk-2} .........

[@AS-gv1on]

Hello, Is this possible that we can configure a two-node Elasticsearch cluster as part of a disaster recovery (DR) strategy?

[@LabITtraining]

it is recommended to have at least a three node cluster for DR.

[@emrahkzltan9798]

Hi thanks for video it is very helpful but i have some issues while using virtualbox as node in my local. In my setup there are 3 node elk and one node kibana. All of them in seperate virtualbox instance. I am following all your steps but i am getting this error: java .secuirty.unrecoverablekeyexception: failed to decrypt safe contents entry: javax.crypto.BadPaddingException: given final block not properly padded. NOTE: i didnt add any password in any cert or other. Same with your steps. Could you please help me?

[@LabITtraining]

Looks like something’s wrong with your certificates

[@akhilkella3642]

Is it okay to make 3 nodes in cluster and kibana on first node

[@LabITtraining]

Yes you can

[@muhibali8876]

Hi Thanks for the awesome video. But elastic_multinode_install.md does not exist in your gitlab repo. Could you please upload it...

[@LabITtraining]

i changed the structure in my repository, i will update the descriptions tab.

[@LabITtraining]

updated the link, thanks for pointing it out.

[@akhilkella3642]

I followed the same process yet curl shows number of nodes= 1 Useless of trying this process Elastic search doesn't even run in browser

[@LabITtraining]

That means your nodes are not able to join the cluster , did you setup your cluster name correctly ?