AWS Elasticsearch Service | Manual Snapshots | Backup to S3 | Index Migration

[@AnkitAnchan666]

Thank you so much for this video. It was difficult to understand all these steps from AWS documentation, but your video made it pretty clear. Keep up the good work, Cheers!

[@ValaxyTechnologies]

You are welcome.

[@jurkinss1]

thanks man. Helped even 5 years after !

[@mateoloaizarios2476]

I already solve the vast majority of my problems. Thank you very much for the help! That good video.

[@ValaxyTechnologies]

Great. Do you think any documentation/script should be updated to make it any easier. What is your suggestions.

[@mateoloaizarios2476]

@@ValaxyTechnologies I do not think so, your explanation is excellent. I save a lot of work! What I think you need to complement is that at the time of one load the AWS CLI must be defined with the command "aws configure" what are the variables aws_access_key_id and aws_secret_access_key. With this I already worked the doubts that you did. Complement it with this and I think that the viewers will not have a greater conflict with your video! (Y)

[@rafarodriguez6392]

Thank you very much from Spain!

[@ADV-IT]

Thank you Man! You saved me

[@ValaxyTechnologies]

So you have seen the ghost(of the need to take manual snapshots) in your work?

[@ADV-IT]

@@ValaxyTechnologies Yeah, need to snapshot Elasticsearch cluster every 1 hour to S3

[@mylifetales07]

Hii .I am doing the same process but facing error in connection .kindly help me in this...thabnk you

[@TAICHI1SCO]

another video with great substance. Thanks for sharing

[@TheFlareMind]

Is it mandatory to create an IAM User and attach the Policy (Containing the iam:passrole and EsHttpPut) to the user? Is it possible to create an IAM Role and attach the Policy (Containing the iam:passrole and EsHttpPut) to the Role so this Role can be assumed by another AWS entity?

[@halilulrahamane8798]

Good one. How to list aws elastic search snapshot repository ? We have few of ES but not sure backup repo is configured or not.

[@shivshankarb]

Hi there, thank you for video looks good to me.. just a question, can't we use the same role(my-manual-es-snap-creator-role) to assume by ec2 used to register repo with out having to configure aws user? I can't get my head around with requirement to create user. Regards, Shiv

[@makeitsimple3337]

Can't we do these through cloud watch event with lambda . Because it is taking lot of manual steps here

[@vrakash1]

Thank You... Presented well 👌

[@ValaxyTechnologies]

So nice of you

[@Bbb-qh2dj]

Thank you so much! This is a super helpful video!

[@machicloud9528]

Thanks! much @Valaxy Tech.. can we migrate ES Between two AWS accounts.

[@mylifetales07]

503 { "message": "No server available to handle the request", } I am getting this error Can any one help me with this?

[@prabhujseelan]

Thank u so much brother... It's perfectly worked ... u saved my day..

[@ValaxyTechnologies]

Awesome. Good to hear.

[@deepeshsee]

@Valaxy Technologies Hi.. Great tutorial, I was looking for a solution to the auth error I was receiving. I have a limitation in executing the backup as an IAM user, instead of that is it possible grant those roles to the master user created to access kibana dashboard. i tried with creating a role with the iam:passrole access and assigned that role under the role mappings in Kibana, but getting "User: anonymous is not authorized to perform: iam:PassRole...

[@maddai1764]

awesome video. thx from senegal

[@ValaxyTechnologies]

Welcome to #Valaxy Channel. Dont forget to like & subscribe :)

[@SureshKumar-lo8ln]

Hi, which profile name we configured AWS cli.?? We will configure the same es region ec2 instance.

[@ValaxyTechnologies]

Yep

[@chintalapativenkataramarahul]

Where will the snapshot repo be stored? Can I see it anywhere?

[@premkumarmani1380]

can we retaining Logs and possibly compressing and shipping to s3 ?

[@habtamuasfaw2062]

First of all thank you for your clear instruction y video and my question is, how would you differentiate which induce from es domain and s3 snapshot has taken? I am seeing random induce file name on s3 but on es each induces has their unique name.

[@habtamuasfaw2062]

and I wish you have next p2 y. video how someone can migrating data or this taken snapshot to a different domain in same or different region? basically backup snapshot from one es and restore to another es cluster? thank you

[@ramdevlinguberi8157]

Hi sir, I need assistance on this. Can we apply retention at index level or is it possible at data in index level, suppose I want the last 10 days log, data beyond 10 days should be deleted. Is it possible in a single index.

[@RahulRavichandrannala]

Hi Im getting repository_missing_exception" what can be done here

[@sheetalnainwal3704]

I dont know if i missed something but i really wanted to know why have we used both s3 and repository?

[@tirushv9681]

How can we automate by sending a particular logs after like 30 days to s3

[@jorgeu111]

end up here because the console doesn't list elasticsearch service for the rule creation. I see the trick is selecting any service and then changing it. I think its a bug in AWS but what do I know

[@omen8523]

Instead of running it in an EC2 and creating an IAM user, can I create an IAM role with the same policy as the one attached to the IAM User, attach it to lambda and run the same code? (making sure lambda and ES are in same VPC and subnets?)

[@JasonLehrhoff]

I'm using this as a Lambda that can assume the snapshot role in his documentation works fine

[@charliejack3194]

Hi, While performing the registration, I am getting the following error as mentioned below : {"error":{"root_cause":[{"type":"parse_exception","reason":"unknown key [type] for create index"}],"type":"parse_exception","reason":"unknown key [type] for create index"},"status":400} Looking forward to your response.

[@mohammednishad9734]

Thank you so much. can we automate the backup

[@ValaxyTechnologies]

Yes, you can

[@Duminda79]

HI I need to backup single indices how do I do that ?

[@vinodv2206]

Thanks for a nice video. But I am getting the 400 error while registering the repo. Error: {"error":{"root_cause":[{"type":"invalid_index_name_exception","reason":"Invalid index name [], must not be empty","index":"","index_uuid":"_na_"}],"type":"invalid_index_name_exception","reason":"Invalid index name [], must not be empty","index":"","index_uuid":"_na_"},"status":400}. any advice, Thanks in advance

[@hemanth_byra]

Thank you so much for your video. I'm facing some issue, my index document count is very large it's around 80lakh, when I execute the command to take snapshot it will work for just few minutes and the result wil be successful with only 20 documents. Kindly help me how to take snapshot of the entire index.

[@mateoloaizarios2476]

Where you enter in the minute @5:03? I have not been able to continue from this step. I do not know what to do

[@ValaxyTechnologies]

If you have AWS CLI configured, then update the host for the file along with the region, Post your error, if you still have issues.

[@amargujeti6194]

In s3 bucket why it is not showing in format of snapshot name which is you provide while taking snapshot

[@ValaxyTechnologies]

How is that a problem? [Pets vs Cattle]Snapshots are Pets, dont worry about fancy names, if it is not a problem

[@Gabu363]

Can i get the document for this video?

[@pramodkati561]

Very well done!!

[@ValaxyTechnologies]

Thank you very much!

[@ramakrishnathotakura4063]

@here I was getting below error when I was running curl command for the snapshot. {"Message":"User: anonymous is not authorized to perform: es:ESHttpGet"}

[@dharmrajodiya7950]

I am getting same error and looks this is dead end. If anyone has any solution please share.

[@JasonLehrhoff]

Only way curl command would work is if you set the domain access policy to 100% open (not good for production!). This is because curl doesn't send the requests signed which is what is required. Instead, I used a Lambda in the same manner that assumes that role and added the role to access policy. VOILA

[@bharathreddy4448]

Hi, can you please do a video of how can we delete the indexes after moving to S3 by using lamda function and cloudwatch trigger

[@ValaxyTechnologies]

Have you subscribed to #Valaxy channel yet? Have a look here - kzbin.info/door/_evcfxhjjui5hChhLE08tQsearch?query=elasticsearch (it includes deletion as well :) Happy Learning )

[@caseygoodrich7435]

Update the github article without the leading spaces.

[@ValaxyTechnologies]

Is it possible to send a pull request? As a community we all can learn from each other. Thank you.

[@achintamghoshdostidar4658]

Thanks for sharing, link to the document please.

[@ValaxyTechnologies]

Have added the github link now.

[@sudheerkethireddy2176]

Thank you for sharing the video. It is really helpful. I was able to successfully able to register manual snapshot repository. I am running into issue when i am trying to create snapshot {"Message":"User: anonymous is not authorized to perform: es:ESHttpPut"}. can you please help me.

[@ValaxyTechnologies]

Check out ES Resource Access policy - It must be inside a vpc ( you may need a bastion to connect to it, with the APPROPRIATE IAM Role to connect to ES)

[@youcouldbemyonlystar5604]

@@ValaxyTechnologies Hi, im using an old ES that was not connected to a VPC, what can be the solution? :)

[@youcouldbemyonlystar5604]

Hi have you solved this issue?

[@anshukumarsoni2361]

Thanks for this video. I want to take snapshot backup from Elastic search service from AWS and restore to the EC2 nodes based Elasticsearch cluster. Will it work in the same way. if version are same.

[@ValaxyTechnologies]

Sure, why not it is the same index - You just have to modify to use the correct API endpoint on EC2

[@mylifetales07]

Hii.Did it work when you followed the same process in the video.Please hel me I am facing issue .Kindly please help me thank you

[@hemanth_byra]

@anshu kumar soni were you able to achieve it? If so please can you help.

[@nidhimanshu07]

my instance elasticsearch is in localhost so, can we migrate to aws elasticsearch service...

[@ValaxyTechnologies]

Yes, S3 > ES

[@nidhimanshu07]

@@ValaxyTechnologies you are saying that first we have to migrate localhost indexes to S3 and we can import to ES

[@ValaxyTechnologies]

Yep!, much easier that way.

[@bharathreddy5582]

I'm doing on AWS , is it required to do 4 th step in AWS also

[@ValaxyTechnologies]

I did not memorise the steps, what are you trying to do? or NOT do?

[@bharathreddy5582]

@@ValaxyTechnologies I had created a domain in AWS Elastic Service , where I need to move the data in es to S3 bucket.

[@bharathreddy5582]

@@ValaxyTechnologies I'm not getting two polices under the user like Elasticsearch service and IAM. Im getting oly IAM policy

[@ValaxyTechnologies]

At which point[time] in the video, you are having issues? Did you complete ALL the steps mentioned in docs or you skipped any of them?

[@bharathreddy5582]

@@ValaxyTechnologies I got that,but after that while running register.repo.py getting error like requests.exceptions.connectionerror: Https connectionpool(host='esdomain url',port=443, max retries exceeded with url: /_snapshot/my snapshot -repo( caused by newconnectionerror('urllib3.connection.verifiedHTTPSconnection object at 0x103bb3110> : failed to establish a new connection: error 60: operation timed out

[@TOP10ZZ]

I have completed the snapshot process, how do I restore onto a different Elasticsearch domain? (not from the one from where the snapshot is taken)

[@ValaxyTechnologies]

Just change the domain url

[@TOP10ZZ]

@@ValaxyTechnologies getting the following error - [root@es-restore ~]# curl -XPOST 'vpc-es-restore-001-gy67u2lapnkf2ecdwc2e.us-west-1.es.amazonaws.com/_snapshot/my-snapshot-repo/2020-02-28/_restore' {"error":{"root_cause":[{"type":"repository_missing_exception","reason":"[my-snapshot-repo] missing"}],"type":"repository_missing_exception","reason":"[my-snapshot-repo] missing"},"status":404} whereas the repo and the directory does exist.

[@ValaxyTechnologies]

did you create the repo before attempting to restore?

[@TOP10ZZ]

Yes I fixed this error. Needed to register snapshot to both elastic search domains

[@bharathreddy4448]

Index names are different in S3 after taking backup ,how can I find exact index which I'm looking for

[@ValaxyTechnologies]

Modify the script to suit your needs.

[@mcgizmo]

you good!!!

[@ValaxyTechnologies]

We good!

[@mateoloaizarios2476]

how can I do the minute @7:12 in windows?

[@ValaxyTechnologies]

The cat command creates the file, @kzbin.info/www/bejne/fJ6vnqysfZ1mhMk this is shown

[@chomskyhitchens]

I just found this totally confusing - left out too many steps :(

[@ValaxyTechnologies]

Did you try following the entire series ( If you already didn't have experience with ELK stack ) listed in the description? Also can you let us know what was missed out?