Day-4 | Terraform State DeepDive | Remote Backend | State Locking

[@madhusaisomarathi]

Hi Bro. Thanks anna for the free courses , i got a good job in one of the MNC company as senior Cloud infrastructure as service role. If every person in IT dedicates some free time for others instead focusing on scraping money from others for time, no one would be jobless. Your initiative is helping others including their dependent families, no other employee would do same as you. Thanks anna.

[@AbhishekVeeramalla]

Hearty Congratulations 🎉 .. Wish you all the best.

[@MHb789]

Did you have any previous experience in IT field?

[@nileshsinha957]

🎯 Key Takeaways for quick navigation: 01:42 📋 *State file in Terraform is where Terraform records the information of the infrastructure it has created.* 03:29 📋 *State file is important because it helps Terraform update and manage existing infrastructure instead of creating new instances or resources.* 07:36 📋 *Advantages of using the state file include updating existing infrastructure and destroying infrastructure in a controlled manner.* 08:05 📋 *One drawback of the state file is that it records sensitive information, such as passwords, by default.* 09:17 📋 *Storing the state file on a personal laptop or any vulnerable device can compromise the security of sensitive information in the state file.* 10:23 🛡️ *Real-world scenarios involve multiple DevOps engineers collaborating on a Terraform project.* 12:12 🛡️ *Storing the entire Terraform logic, including the state file, in a Version Control System can compromise sensitive information.* 14:31 🛡️ *If the state file is not pushed to the Version Control System, Terraform may not know about updates and can delete the corresponding infrastructure.* 16:35 🛡️ *Using a remote backend solves the drawbacks of storing the state file in a Version Control System.* 19:04 🛡️ *With a remote backend, the state file is automatically updated in the external resource (e.g., S3 bucket) when applying Terraform configurations.* 20:41 🛡️ *Storing the state file in a separate remote backend, such as an S3 bucket, enhances security and prevents sensitive information from being exposed in a Version Control System.* 21:09 🛡️ *Multiple remote backends can be used, such as S3 or Azure storage, depending on the cloud provider being used.* 23:13 🛡️ *With a remote backend workflow, DevOps engineers can clone the GitHub repository, make changes locally, verify them with `terraform apply`, and raise a pull request to update the remote state file stored in the S3 bucket.* 27:30 📂 *Before running `terraform init`, there will be no state file available. After running `terraform apply`, the state file will be created and populated with resource information.* 29:03 📂 *The populated state file contains information about the created resources, such as the instance ID, public and private IP addresses.* 30:46 🛡️ *Storing the state file in a Version Control System is not recommended due to security risks and the potential for forgetting to push updates to the state file.* 31:16 🛡️ *Terraform can show the state file using the `terraform show` command.* 31:29 📂 *Deleting the state file can cause Terraform to lose track of the created resources.* 32:10 🛡️ *Using a remote backend like S3 requires creating a `backend.tf` file with the necessary backend configuration.* 33:24 🛡️ *Copying and pasting backend configuration from the HashiCorp documentation or using extensions can simplify the process.* 35:16 🛡️ *S3 bucket creation can be done manually or with Terraform, ensuring unique bucket names.* 37:02 📂 *The S3 bucket name and region are required in the backend configuration.* 38:38 🛡️ *Initializing the project with the backend configuration will create the S3 bucket.* 40:29 📂 *The state file will be created and stored in the S3 bucket instead of locally.* 41:37 📂 *The state file can be stored in an S3 bucket instead of locally.* 42:09 📂 *Using a remote backend allows storing the state file in S3 and accessing it from there.* 42:38 📏 *Modifying the code no longer requires handling the state file manually.* 44:13 🛡️ *Terraform uses locking mechanism to prevent conflicts when multiple people try to update the same project simultaneously.* 45:22 🛡️ *DynamoDB can be used as a locking mechanism for Terraform projects.* 48:15 🛡️ *Adding the DynamoDB lock details to the backend configuration enables Terraform to handle locking.* Made with HARPA AI

[@roshniyadav1539]

Thanks bro it's really helped me a lot I already saw the complete list. just came to revise the concept and your comments save my time.thank u so much

[@EasyTech2000]

Exactly you explain very well like to become zero to hero ☺👍 thanks a lot for such a great inputs and making useful resources... Amazing👍

[@AbhishekVeeramalla]

Thank you 😊

[@ashwinichivate3905]

Gr8 explanation about statefiles and remote backend from end to end.Thanks.

[@AbhishekVeeramalla]

Thanks

[@ArchuWikky]

Hi Abhishek, At the end of this series, please let us know about the common issues/ frequent issues faced in terraform and interview questions too.

[@AbhishekVeeramalla]

For sure

[@AbhishekVeeramalla]

Btw already I am covering a lot things.

[@ArchuWikky]

@@AbhishekVeeramalla 👍 okay

[@getgoingwithdeepak]

Hats-off Abhishek for wonderful explaination and practicals.

[@rehanfazal7354]

thanks GURU-JI for teaching these valuable skills

[@Y.Haritha-ss5jd]

Thanks a lot bro i spend lot of money but no one explained the way u did..You are a gem..may god bless you bro ❤

[@AbhishekVeeramalla]

😍😍😍

[@chukwumaonu7687]

Bro, you made this look so easy and very cool for interview preparation. Thank you so much

[@AbhishekVeeramalla]

You are most welcome

[@sainirmalreddygavini2392]

Really ah wonderful explanation Abhishek , Thankyou much!

[@AbhishekVeeramalla]

Most welcome!

[@rakeshkumarnayak8744]

after seen this video i am first time understood the workflow of statefile and his mechanism.

[@rakeshkumarnayak8744]

Thank you sir.

[@AbhishekVeeramalla]

Thank you so much 😀

[@madhugundarapu]

Thank you Abisheek, For spending your valuable time with us. It will really help a lot of people around the globe

[@dideepakindorkar3413]

Great sir.. neat and clean explanation.. everyone can understand... Thanks 🙏

[@anaramurat733]

Hi! Thanks for free but very valueble information. I have had many technical interviews. Trust me, almost in every interview they asked about terraform state file. " What is the best managent policy for statefile?" "How to keep secure terraform state file?' etc.... Greatings from US.

[@deychand11]

Amazon S3, Azure Storage, Google Cloud Storage are the best locations to keep your storage file encrypted. And make sure that you have proper IAM policies in it

[@awanishaman1612]

Thankyou so much #Abhishek for this valuable tutorial for free it's very informative and easy to understand for me, very helpful 👏👍

[@anilsree06]

Thanks Abhishek!! Able to implement this locking with dynamo db and s3 as remote state backend, It was good hands on practical.

[@AbhishekVeeramalla]

Great 👍

[@msmp4863]

Thank you Abisheek, For spending your valuable time for us. It will really help a lot of peoples in globe

[@AbhishekVeeramalla]

It's my pleasure

[@shubhamkalekar1519]

Hi abhishek Thanks for your all effort, in real time how the resources will create , whether applying infrastructure using apply command or they use pipeline for infra creation.... If yes then which case we need to create pipeline ams

[@dileepkumarbandi8066]

Great session Bayya 👌..Previously I had listened terraform from two resources but still lot of confusion on these state and remote backends ...Now everything is crystal clear ...Thanks Bayya

[@AbhishekVeeramalla]

Welcome 🙏

[@fritzdamiendengambog9169]

Guys do not forget to always hit the like button. And subscribe. Hi bro @AbhishekVeeramalla greetings from Africa Cameroon

[@AbhishekVeeramalla]

😍😍😍

[@hungryman6711]

absolutely a master class ⭐⭐⭐⭐⭐

[@Pardeep_Sharma999]

Great session ,Thanks Abhishek for your great work

[@AbhishekVeeramalla]

Welcome

[@manjeshsno596]

Hurray, I was waiting for this video and completed with good understanding Sir..thank you so much for your time.🎉

[@AbhishekVeeramalla]

You are most welcome

[@Shweta.Gaikwad1966]

Thanks for the fantastic video-your clear explanations made everything so easy to understand! 👏

[@raghava17]

Hi @ Abhishek sir, Great explanation, Actually im azure guy and i watched all the playlists and i implemented same in Azure Cloud , i got expected Results without any issues while implementing things on azure using terraform backends and modules and with using variables , by saying Thank's is very small word for the wonderful content you delivered 😍✌ 🎉🎉 I'm very happy while implementing same things on Azure by getting expected Results with out Any Errors 🤩🤩🤩,,, Thanks a lot sir 🥳🥳🥳

[@dileepkumarbandi8066]

@raghava17 bro do u have any real time azure ci-cd projects ?

[@AbhishekVeeramalla]

Thanks a lot

[@ojosamuel3777]

Great Work @Abhishek. really a refresher course for me. Hope you will share with us the case of using Terraform cloud as remote backend

[@AbhishekVeeramalla]

Thank you so much 😀

[@entertainment_show9597]

Hi Abhishek sir, is it possible to use another database resource for the locking system instead of DynamoDB? Or is DynamoDB the best option for the locking system?

[@sandeepsinghrathore505]

@Abhishek.Veeramalla - Super very clear..

[@rekhadevikondeti5764]

Beautifully explained 👏

[@AbhishekVeeramalla]

Thank you 🙂

[@AtulMisal96]

Hi Abhishek Bhaiya.. At the end of this series please make one video on end to end project on terraform like creation of multiple AWS services...

[@AbhishekVeeramalla]

For sure

[@harshnegi200]

Completed..! Thanks Abhishek.

[@ajitsahoo4875]

Hi Abhishek thanks for the lesson. Also we can remove the lock file forcible in some scenarios. Right?

[@jaswanth_kumar_k]

Thanks for the consistency and efforts ❤

[@AbhishekVeeramalla]

Welcome

[@nehaverma-xn3ed]

Hi, Great video and explanation, one quick question is it good to have aws s3 configuration and resource information in same repo ?? beacause while destroying it was giving error and it should not delete s3 bucket as it consist state file.

[@mahendrapalla1373]

Hi @abhishek, The way you explaining the content is really good and must appreciate. at the END of this terraform course, are you going to provide video to set-up production ready EKS cluster in AWS for our understandings ? Thank you, Mahendra.

[@AbhishekVeeramalla]

Sure, I will try my best

[@sthitaprangyachhatoi4967]

Hey Abhishek, Please consider creating one Production kind of eks cluster setup using terraform and using all the concepts we are learning in this series.. That would be really helpful to get idea on where to use what i believe.

[@AbhishekVeeramalla]

will try

[@PY-D]

Good idea.

[@pritamburnwal6706]

Hi Abhishek, Thanks for the amazing videos of terraform. I am learning a lot with that. at the time stamp of 19:50, the solution for the second issue with local state file I got that, but can you explain a little bit more what exactly the problem is. I got little confused with that issue.

[@sadasivagaming]

good explaination

[@nyeemTube11]

Hello Abhishek thanks for the Zero to Hero Terraform Series. Could you please include a few real-life challenges which could be very beneficial for all people who are learning from you.

[@mayurichikne526]

Hi sir, one request... By the end of series... Plz make full project of how to create real-time used vpc ,subnet in Aws infra using tf.. Thanks in advance

[@AbhishekVeeramalla]

sure, will try

[@Jay-yu4yv]

As the other engineers make new changes, statefile will keep on updating. We can enable s3 versioning and can revert back if someone has made some incorrect configurations with the infra. Am I right @Abhishek.Veeramalla??

[@InderjeetYadav-f1s]

@abhishek, for eg, we created multiple resources using terraform like multiple ec2, vc, sg. I we can delete a specific resource using terraform destroy command? if we run terraform destroy will it destroy everything which was created previously ? Please help me to understand

[@ramthumati2620]

nice brother very understand

[@adarshjha__1]

Day 4, lesss gooo! 🙌🔥

[@AbhishekVeeramalla]

Thanks

[@srivanikoppula]

thank uu bro for this seesion very neat explanation and pratical too

[@vasu-if3de]

Hi Abhiske, I hvae few doubts on terraform 1. how can we give the IAM policy to the S3 and instances 2. Can you please explain the how can we create the VPC, Subnets with CIDR . CAN YOU MAKE A VIDEO .

[@ChifoAndrew]

Hello Abhishek, pls how can the lock policy be updated? Is it done on the AWS console or can it be provisioned through terraform?

[@mohankumarr1432]

Thank you so much Abhishek ❤😊

[@AbhishekVeeramalla]

Thank you so much 😀

[@Maharaja98Cubic]

Hi Abhishek, your content and method of teaching is good and i personally like that style, so thank you. I have one questions on Remote Backend, what if there are two Devops person access the same file at the same time and during update there could be a chance that second person could miss the update or over ride first persons changes. Is that possible and if YES then what is the solution?

[@AbhishekVeeramalla]

You can implement the locking using state locking. DymamoDB can be used for that

[@tuonimihart2009]

youre a great teacher

[@AbhishekVeeramalla]

Thanks a lot !!

[@sagarahire-ys3uj]

Amazing content as always...forget ppl who r preparing for interviews even working devops ppl are also learning from ur videos that much of relevance is there in ur teaching...i have one question...what is the default behaviour of terraform when we dont create dynamodb table and state lock? how the situation is different before applying dynamodb state lock configuration and aftermath of it?

[@aishwaryap.s.v.s7387]

hi annaya !!! actually i paid for courses but ur courses are more than those courses .i watched terraform in one stretch and understood the concepts!!! thanks anna for ur help!!!

[@AbhishekVeeramalla]

All the best

[@handsomevillian]

Hello Sir, Can we configure NFS mount as a remote backend?

[@agun21st]

I am going to master in Terraform by you sir. 😇

[@AbhishekVeeramalla]

😍

[@proplusgaming3361]

00:06 Terraform state file and backend 02:43 Using the State file in Terraform 07:36 Advantages and disadvantages of using the state file in Terraform. 10:07 Sensitive information in state files and potential risks 14:59 Using remote backend in terraform to fix the drawbacks of state files 17:19 Store terraform state file in S3 bucket for security and easy access 21:37 Explaining the workflow of updating AWS resources using Terraform in interviews 23:55 Configuring Terraform and AWS in Code Spaces environment 28:12 Executing the terraform apply command creates a state file and updates it with the resources being created. 30:19 Creating a Version Control System-supported project in Terraform 34:35 To create a bucket in AWS S3, you can use Terraform or create it manually. 36:45 Create a unique AWS S3 bucket for Terraform 41:37 Terraform project modified to store state file on S3 bucket 43:48 To avoid conflicts in the terraform project, locking mechanism is important 47:58 Implement backend mechanism with DynamoDB locking. 50:05 This is a simple code that can be executed with the help of documentation and the visual studio extension.

[@vikramkamble5330]

Best man for devops🙌

[@AbhishekVeeramalla]

😍

[@Jalal921]

Very good explanation Abhi

[@AbhishekVeeramalla]

Thanks a lot

[@aravindsuresh5848]

@ 42:43 As you mentioned no one can access the statefile but when a user execute the command terraform show, it will show the statefile details so anyway the user can see the statefile details right? Got this question, am I missing anything?

[@askaralil]

Will terraform show displays sensitive information of state file from s3 bucket of remote backend?

[@ketanmorey1953]

very good series Abhishek sir❤ if we are following this to stored state file in s3 and locked it with dynamodb, if new member of terraform team wants to access state file , then how he access it.. please explain this sir.. Thank you🙏

[@ghareeshkumar]

Hi Abhishek, Is subnet ID and ami should be belonga to same 'EC2 instance'?

[@pavankumardasari6606]

One doubt, We are using S3 to store state file and dynamo db to maintain locking, Does state file also will be stored in Dynamo db or we are using dynamo db only for locking. Just we clarification on dynamo db some more.

[@AbhishekVeeramalla]

Only locking

[@pavankumardasari6606]

@@AbhishekVeeramalla Got it, Thanks

[@ashwinswain1649]

Hi Abhishek, The lock file is pushed to remote n not on local only so it will be able to use lock mechanism if simultaneously t-apply is done? Also in our project we use terragrunt and the explanation is given by the architect on his LinkedIn post i wish you code read n explain it to me in simple words..i cant paste screenshot here but he has written about backend cant be modified without init and that terragrunt allows module level statefile

[@karthikkumar4233]

Hi Anna, for dynamodb_table why did you use this step--> billing_mode = “PAY PER REQUEST”? If it is mandatory step how does this help in cost optimization?

[@durgaganesh311]

Hello abhishek ! As we are deleting the state file on the first apply then how we can destroy the created s3 and dynamodb without statefile? and the second doubt is there are two instances are creating rather than one ...can you please explain ?😔

[@faisalraj6654]

Hi Abhishek, in timestamp@42:00, when you say, no one will have access to state file/backend.tf, but when devOps is creating a resource, wouldn't the state file required and visible at the root of the repository? Terraform must know the presence of the state file to avoid duplication of the resource. Correct?

[@sparun1607]

As suggested by terraform, can we go with option suggested ? will solve if we miss configured ? Initializing the backend... Error: Backend configuration changed A change in the backend configuration has been detected, which may require migrating existing state. 1. If you wish to attempt automatic migration of the state, use "terraform init -migrate -state" 2. If you wish to store the current configuration with no changes to the state, use "terraform init -reconfigure".

[@pavithrang6760]

Thanks for ur video bro. It helps me a lot to understand about terraform state file. Actually I've one doubt in backend.tf file. When I configure backend in terraform, state file successfully moved to s3 bucket but in terraform file terraform.tfstate.backup file is now created which holds all information about resource. How do I remove this file? Plz provide some tips.

[@singhbrozz1996]

awesome explainnation 😊 . can we store state file in database?

[@sowmithrawulwar2823]

You could've explained this backend concept using modules. Folder structure is confusing and what files to create.

[@smartsiddhu30]

Hi bro, Thanks it was a wonderful section, I have tried implementing the lock and backend state, but I could see a file called terraform.tfstate.backup after using the backend as s3. As it contains the same information, Is there is any to block that, Am I missing something!!

[@vinaykpise7998]

Question: I have already provisioned the ec2 resource using terraform later i will update tf file with user data, will this change in ec2 instance.

[@AshokSharma-xv2ks]

To put the state file in s3 we need to first delete the state file we can't able to do in first time ?

[@brownpanther5280]

When we moved state file to S3 and linked locked file to Dyanamo DB , my assumption was .terraform.lock.hcl shouldn't be created in local as locking is taken care by Dynamodb (Like the way i dont see state file ln local ) . However , I do see lock file in local . I created new folder and tried doing same , still lock file is getting created in local . -- Can you help understand this ?Im i missing something?

[@ashishpandey8766]

Thanks Abhishek

[@AbhishekVeeramalla]

Most welcome

[@abhishekkulkarni3466]

Hi Abhishek, first of all thanks for amazing series. I have few doubts in this video. 1) I don’t understand the flow how statefile is being stored in S3 bucket. You said when we run terraform apply then statefile is generated in s3 bucket. But what if I run terraform apply and then my PR got rejected? Basically I wanted to know how this remote backend mechanism will work in case of feature branches? 2) What if I just wanted to try out some things and run terraform apply several times. How statefile is stored in that particular scenario?

[@rameshmanchala-o4d]

Hi 1. if the Pr is rejected and you make necessary changes and do terraform apply, state file will also get updated ryt. You don't need to worry what is happening in state file. It will be updated accordingly. 2. any number of times you try to apply state file will update the same. So as mentioned before updating state file is something terraform will take care. We don't need to get worried

[@sasmitamallia6960]

what to do if we have to download the Huge size State file that is present in S3 or Azure Storage Account ? apart from downloading manually from bucket ? Do we have any other options?

[@devopsTox-bb2cr]

Hi, As the state file contains sensitive data and it will be in s3. What about the file providers.tf which contains aws_secret and aws_key ? How can I secure it ? Can I create it and after the first terraform apply command, delete it ?

[@ravitejadarla3882]

Hi Abhishek, I request you to make an in-depth video on the Kubernetes cluster step-up on the Ubuntu ec2 Instance there is no proper video on KZbin.

[@MyGiri123]

Can you please explain what happens internally when we issue terraform init command ?

[@chowdarybvsn]

In real time, are we using terraform for complete setup in AWS? or only for major resources creation only?

[@AbhishekVeeramalla]

Complete

[@vikkivikas3685]

don't delete the terraform.tfstate file, it gives bucket already exists, but it working fine. Use terraform init directly without deleting terraform.tfstate file, it will not give any error, I am giving just for information

[@rengaraj98]

Hi Abhishek, Kind request. Please answer the below common question which i encountered in most of interviews. We have created the AWS resource for ex an ec2 instance through terraform, and then the state file gets deleted. If we reapply the same terraform code for recreating the ec2 instance. what will happen, will the ec2 instance will be created or not or there will be any errors.

[@AbhishekVeeramalla]

I did the same thing in the video. Please check

[@rengaraj98]

Thanks for the reply@@AbhishekVeeramalla from the video you have explained, like if state file is not there and we create the resource again with minor change it will recreate the whole resource from scratch. But what happens when state file is not there and we re apply the same code without any change. My understanding is that it will create the resource with different ID, Is that correct? please clarify?

[@lingrajbiradar4416]

yes, it will create the resource with diff ID@@rengaraj98

[@nick-sx2zn]

is there any mechanism to execute the resources and files order wise, instead of deleting the state file multiple times

[@kittubhai4799]

Hai Abhishek small question For ami id and subnet id we have see aws UI only, without opening can't we do ??

[@cliftonavil]

Excellent 😇

[@PremKumar-co5ym]

Hi @ Abhishek sir At end of the series plz give some interview questions and answers.

[@AbhishekVeeramalla]

sure

[@howtotelevision]

❤ love it

[@AbhishekVeeramalla]

Thanks

[@ravipanavi]

Hi abhi, If we are required to work with multiple people will work the same project, then what is the option.

[@AbhishekVeeramalla]

Thats why you store terraform files in git

[@prasadreddy8360]

Super sir

[@AbhishekVeeramalla]

Thanks

[@ManojGandham-dw2xy]

well explained💯

[@Atvkaur1111]

Sir do we need to have coding (development)knowledge for devops role?

[@ashwinreddy2866]

Hi Abhishek, I have doubt at 20:54 , What if DevOps engineer pushes the Terraform code with out executing it.? He updated the infrastructure but didn't execute "Terraform apply" and pushed it to repo. what happens in that case? Does it lies in the hands of PR reviewer to check ?

[@SriGowthamNallasamy]

Hi Abhishek - If we are using the remote back-end also the terraform show command reveals the contents of the state file . How can we secure it in this scenario

[@AbhishekVeeramalla]

Thats where your IAM policy comes in to picture. You can also use S3 bucket policies

[@SriGowthamNallasamy]

@@AbhishekVeeramalla Thank You Abhishek ❤️🙏

[@SurajM-qs3wx]

instead of Dynamo db can we use Mysql??

[@gajananwali9705]

Did hands-on on this. And at last tried deleting all resources. But it is showing as destroy complete. But I can see ec2 instance and S3 bucket still running. May I know why

[@AbhishekVeeramalla]

It should not be the case. Please check your state file. Everything thats in statefile will be deleted

[@FreakinTraveller]

How do we remove lock from remote state file from dynamodb lock table?

[@charansai9891]

Hi Abhishek. Initially you have created S3 bucket and later we deleted statefile. then again we tried to create S3 bucket with same name and configuration with out state file. and we used remote backend to store state file in S3. But if we use same name and configuration which already used we'll get error right. stating Resource already exists. Can you explain on this.

[@m2abhishek836]

That's correct, he also got the error. See 41:58. But he immediately "clear"ed the screen to escape it. Sometimes these things happens.😀

[@shadowlors13]

Yes, he skiped it. instead of deleting the state file we could have just kept it in the local and use git ignore or something like that to not push the state file to git or any repository and from backend push the state file to s3 bucket. this way it would just push the backend file to s3 and we will not get the s3 error and no extra instance will be created.

[@umashanker3244]

Hi Abhishek please make interview questions on all devops tools and AWS cloud

[@AbhishekVeeramalla]

Sure

[@umashanker3244]

Hearty Congratulations Abhishek for your newyork poster

[@manid2900]

Hi Anna...today are you posting the day-5 vedio?

[@AbhishekVeeramalla]

Its already uploaded