Day-4 | Terraform State DeepDive | Remote Backend | State Locking

[@madhusaisomarathi]

Hi Bro. Thanks anna for the free courses , i got a good job in one of the MNC company as senior Cloud infrastructure as service role. If every person in IT dedicates some free time for others instead focusing on scraping money from others for time, no one would be jobless. Your initiative is helping others including their dependent families, no other employee would do same as you. Thanks anna.

[@AbhishekVeeramalla]

Hearty Congratulations 🎉 .. Wish you all the best.

[@MHb789]

Did you have any previous experience in IT field?

[@nileshsinha957]

🎯 Key Takeaways for quick navigation: 01:42 📋 *State file in Terraform is where Terraform records the information of the infrastructure it has created.* 03:29 📋 *State file is important because it helps Terraform update and manage existing infrastructure instead of creating new instances or resources.* 07:36 📋 *Advantages of using the state file include updating existing infrastructure and destroying infrastructure in a controlled manner.* 08:05 📋 *One drawback of the state file is that it records sensitive information, such as passwords, by default.* 09:17 📋 *Storing the state file on a personal laptop or any vulnerable device can compromise the security of sensitive information in the state file.* 10:23 🛡️ *Real-world scenarios involve multiple DevOps engineers collaborating on a Terraform project.* 12:12 🛡️ *Storing the entire Terraform logic, including the state file, in a Version Control System can compromise sensitive information.* 14:31 🛡️ *If the state file is not pushed to the Version Control System, Terraform may not know about updates and can delete the corresponding infrastructure.* 16:35 🛡️ *Using a remote backend solves the drawbacks of storing the state file in a Version Control System.* 19:04 🛡️ *With a remote backend, the state file is automatically updated in the external resource (e.g., S3 bucket) when applying Terraform configurations.* 20:41 🛡️ *Storing the state file in a separate remote backend, such as an S3 bucket, enhances security and prevents sensitive information from being exposed in a Version Control System.* 21:09 🛡️ *Multiple remote backends can be used, such as S3 or Azure storage, depending on the cloud provider being used.* 23:13 🛡️ *With a remote backend workflow, DevOps engineers can clone the GitHub repository, make changes locally, verify them with `terraform apply`, and raise a pull request to update the remote state file stored in the S3 bucket.* 27:30 📂 *Before running `terraform init`, there will be no state file available. After running `terraform apply`, the state file will be created and populated with resource information.* 29:03 📂 *The populated state file contains information about the created resources, such as the instance ID, public and private IP addresses.* 30:46 🛡️ *Storing the state file in a Version Control System is not recommended due to security risks and the potential for forgetting to push updates to the state file.* 31:16 🛡️ *Terraform can show the state file using the `terraform show` command.* 31:29 📂 *Deleting the state file can cause Terraform to lose track of the created resources.* 32:10 🛡️ *Using a remote backend like S3 requires creating a `backend.tf` file with the necessary backend configuration.* 33:24 🛡️ *Copying and pasting backend configuration from the HashiCorp documentation or using extensions can simplify the process.* 35:16 🛡️ *S3 bucket creation can be done manually or with Terraform, ensuring unique bucket names.* 37:02 📂 *The S3 bucket name and region are required in the backend configuration.* 38:38 🛡️ *Initializing the project with the backend configuration will create the S3 bucket.* 40:29 📂 *The state file will be created and stored in the S3 bucket instead of locally.* 41:37 📂 *The state file can be stored in an S3 bucket instead of locally.* 42:09 📂 *Using a remote backend allows storing the state file in S3 and accessing it from there.* 42:38 📏 *Modifying the code no longer requires handling the state file manually.* 44:13 🛡️ *Terraform uses locking mechanism to prevent conflicts when multiple people try to update the same project simultaneously.* 45:22 🛡️ *DynamoDB can be used as a locking mechanism for Terraform projects.* 48:15 🛡️ *Adding the DynamoDB lock details to the backend configuration enables Terraform to handle locking.* Made with HARPA AI

[@roshniyadav1539]

Thanks bro it's really helped me a lot I already saw the complete list. just came to revise the concept and your comments save my time.thank u so much

[@sandeepgsastry4731]

23:13 is it correct that pull request is for updating the s3 bucket ? i believe pull request is for the terraform code in github. The statefile in s3 gets updated automatically

[@EasyTech2000]

Exactly you explain very well like to become zero to hero ☺👍 thanks a lot for such a great inputs and making useful resources... Amazing👍

[@AbhishekVeeramalla]

Thank you 😊

[@ashwinichivate3905]

Gr8 explanation about statefiles and remote backend from end to end.Thanks.

[@AbhishekVeeramalla]

Thanks

[@Y.Haritha-ss5jd]

Thanks a lot bro i spend lot of money but no one explained the way u did..You are a gem..may god bless you bro ❤

[@AbhishekVeeramalla]

😍😍😍

[@getgoingwithdeepak]

Hats-off Abhishek for wonderful explaination and practicals.

[@madhugundarapu]

Thank you Abisheek, For spending your valuable time with us. It will really help a lot of people around the globe

[@sharan.2822]

those who are using the terraform destroy command to delete all the resources which was created using the following video please make sure that the you need to write the main.tf code in a way that the statefile should be first deleted then only we can delete the bucket otherwise the terraform will throw a error for us saying that the bucket is not empty and yeah this is applicable only for this project only. at last thank you abhishek for making the concept crisp and clear😊

[@AbhishekVeeramalla]

Welcome

[@awanishaman1612]

Thankyou so much #Abhishek for this valuable tutorial for free it's very informative and easy to understand for me, very helpful 👏👍

[@rehanfazal7354]

thanks GURU-JI for teaching these valuable skills

[@raghava17]

Hi @ Abhishek sir, Great explanation, Actually im azure guy and i watched all the playlists and i implemented same in Azure Cloud , i got expected Results without any issues while implementing things on azure using terraform backends and modules and with using variables , by saying Thank's is very small word for the wonderful content you delivered 😍✌ 🎉🎉 I'm very happy while implementing same things on Azure by getting expected Results with out Any Errors 🤩🤩🤩,,, Thanks a lot sir 🥳🥳🥳

[@deepufanarmy]

@raghava17 bro do u have any real time azure ci-cd projects ?

[@AbhishekVeeramalla]

Thanks a lot

[@anaramurat733]

Hi! Thanks for free but very valueble information. I have had many technical interviews. Trust me, almost in every interview they asked about terraform state file. " What is the best managent policy for statefile?" "How to keep secure terraform state file?' etc.... Greatings from US.

[@deychand11]

Amazon S3, Azure Storage, Google Cloud Storage are the best locations to keep your storage file encrypted. And make sure that you have proper IAM policies in it

[@chukwumaonu7687]

Bro, you made this look so easy and very cool for interview preparation. Thank you so much

[@AbhishekVeeramalla]

You are most welcome

[@dideepakindorkar3413]

Great sir.. neat and clean explanation.. everyone can understand... Thanks 🙏

[@deepufanarmy]

Great session Bayya 👌..Previously I had listened terraform from two resources but still lot of confusion on these state and remote backends ...Now everything is crystal clear ...Thanks Bayya

[@AbhishekVeeramalla]

Welcome 🙏

[@Vamsi-c5e]

Hi Anna, your real time examples are excellent and Thank you so much

[@sainirmalreddygavini2392]

Really ah wonderful explanation Abhishek , Thankyou much!

[@AbhishekVeeramalla]

Most welcome!

[@Shweta.Gaikwad1966]

Thanks for the fantastic video-your clear explanations made everything so easy to understand! 👏

[@anilsree06]

Thanks Abhishek!! Able to implement this locking with dynamo db and s3 as remote state backend, It was good hands on practical.

[@AbhishekVeeramalla]

Great 👍

[@msmp4863]

Thank you Abisheek, For spending your valuable time for us. It will really help a lot of peoples in globe

[@AbhishekVeeramalla]

It's my pleasure

[@rakeshkumarnayak8744]

after seen this video i am first time understood the workflow of statefile and his mechanism.

[@rakeshkumarnayak8744]

Thank you sir.

[@AbhishekVeeramalla]

Thank you so much 😀

[@manjeshsno596]

Hurray, I was waiting for this video and completed with good understanding Sir..thank you so much for your time.🎉

[@AbhishekVeeramalla]

You are most welcome

[@ojosamuel3777]

Great Work @Abhishek. really a refresher course for me. Hope you will share with us the case of using Terraform cloud as remote backend

[@AbhishekVeeramalla]

Thank you so much 😀

[@ArchuWikky]

Hi Abhishek, At the end of this series, please let us know about the common issues/ frequent issues faced in terraform and interview questions too.

[@AbhishekVeeramalla]

For sure

[@AbhishekVeeramalla]

Btw already I am covering a lot things.

[@ArchuWikky]

@@AbhishekVeeramalla 👍 okay

[@gratefultolife2642]

Excellent video so far !!

[@Pardeep_Sharma999]

Great session ,Thanks Abhishek for your great work

[@AbhishekVeeramalla]

Welcome

[@sandeepsinghrathore505]

@Abhishek.Veeramalla - Super very clear..

[@hungryman6711]

absolutely a master class ⭐⭐⭐⭐⭐

[@harshnegi200]

Completed..! Thanks Abhishek.

[@rekhadevikondeti5764]

Beautifully explained 👏

[@AbhishekVeeramalla]

Thank you 🙂

[@ljfpsgaming]

hi abishek do you have ansible from zero to hero and rancher from zero to hero... would you mind creating a tutorial for us. for the benefit of the others as well.. God Bless you and thank you!

[@srivanikoppula]

thank uu bro for this seesion very neat explanation and pratical too

[@jaswanth_kumar_k]

Thanks for the consistency and efforts ❤

[@AbhishekVeeramalla]

Welcome

[@aishwaryap.s.v.s7387]

hi annaya !!! actually i paid for courses but ur courses are more than those courses .i watched terraform in one stretch and understood the concepts!!! thanks anna for ur help!!!

[@AbhishekVeeramalla]

All the best

[@fritzdamiendengambog9169]

Guys do not forget to always hit the like button. And subscribe. Hi bro @AbhishekVeeramalla greetings from Africa Cameroon

[@AbhishekVeeramalla]

😍😍😍

[@AtulMisal96]

Hi Abhishek Bhaiya.. At the end of this series please make one video on end to end project on terraform like creation of multiple AWS services...

[@AbhishekVeeramalla]

For sure

[@adarshjha__1]

Day 4, lesss gooo! 🙌🔥

[@AbhishekVeeramalla]

Thanks

[@nyeemTube11]

Hello Abhishek thanks for the Zero to Hero Terraform Series. Could you please include a few real-life challenges which could be very beneficial for all people who are learning from you.

[@mohankumarr1432]

Thank you so much Abhishek ❤😊

[@AbhishekVeeramalla]

Thank you so much 😀

[@sadasivagaming]

good explaination

[@vikramkamble5330]

Best man for devops🙌

[@AbhishekVeeramalla]

😍

[@sthitaprangyachhatoi4967]

Hey Abhishek, Please consider creating one Production kind of eks cluster setup using terraform and using all the concepts we are learning in this series.. That would be really helpful to get idea on where to use what i believe.

[@AbhishekVeeramalla]

will try

[@PY-D]

Good idea.

[@ramthumati2620]

nice brother very understand

[@agun21st]

I am going to master in Terraform by you sir. 😇

[@AbhishekVeeramalla]

😍

[@proplusgaming3361]

00:06 Terraform state file and backend 02:43 Using the State file in Terraform 07:36 Advantages and disadvantages of using the state file in Terraform. 10:07 Sensitive information in state files and potential risks 14:59 Using remote backend in terraform to fix the drawbacks of state files 17:19 Store terraform state file in S3 bucket for security and easy access 21:37 Explaining the workflow of updating AWS resources using Terraform in interviews 23:55 Configuring Terraform and AWS in Code Spaces environment 28:12 Executing the terraform apply command creates a state file and updates it with the resources being created. 30:19 Creating a Version Control System-supported project in Terraform 34:35 To create a bucket in AWS S3, you can use Terraform or create it manually. 36:45 Create a unique AWS S3 bucket for Terraform 41:37 Terraform project modified to store state file on S3 bucket 43:48 To avoid conflicts in the terraform project, locking mechanism is important 47:58 Implement backend mechanism with DynamoDB locking. 50:05 This is a simple code that can be executed with the help of documentation and the visual studio extension.

[@tuonimihart2009]

youre a great teacher

[@AbhishekVeeramalla]

Thanks a lot !!

[@mahendrapalla1373]

Hi @abhishek, The way you explaining the content is really good and must appreciate. at the END of this terraform course, are you going to provide video to set-up production ready EKS cluster in AWS for our understandings ? Thank you, Mahendra.

[@AbhishekVeeramalla]

Sure, I will try my best

[@Jalal921]

Very good explanation Abhi

[@AbhishekVeeramalla]

Thanks a lot

[@mayurichikne526]

Hi sir, one request... By the end of series... Plz make full project of how to create real-time used vpc ,subnet in Aws infra using tf.. Thanks in advance

[@AbhishekVeeramalla]

sure, will try

[@ashishpandey8766]

Thanks Abhishek

[@AbhishekVeeramalla]

Most welcome

[@cliftonavil]

Excellent 😇

[@itlearntv]

❤ love it

[@AbhishekVeeramalla]

Thanks

[@pritamburnwal6706]

Hi Abhishek, Thanks for the amazing videos of terraform. I am learning a lot with that. at the time stamp of 19:50, the solution for the second issue with local state file I got that, but can you explain a little bit more what exactly the problem is. I got little confused with that issue.

[@ManojGandham-dw2xy]

well explained💯

[@shubhamkalekar1519]

Hi abhishek Thanks for your all effort, in real time how the resources will create , whether applying infrastructure using apply command or they use pipeline for infra creation.... If yes then which case we need to create pipeline ams

[@light-sg5sl]

Sir, where did I make mistake please tell 49:00 I created first main.tf with dynamodb table then I updated in backend.tf but got the same error

[@aravindsuresh5848]

@ 42:43 As you mentioned no one can access the statefile but when a user execute the command terraform show, it will show the statefile details so anyway the user can see the statefile details right? Got this question, am I missing anything?

[@askaralil]

Will terraform show displays sensitive information of state file from s3 bucket of remote backend?

[@FreakinTraveller]

How do we remove lock from remote state file from dynamodb lock table?

[@entertainment_show9597]

Hi Abhishek sir, is it possible to use another database resource for the locking system instead of DynamoDB? Or is DynamoDB the best option for the locking system?

[@sagarahire-ys3uj]

Amazing content as always...forget ppl who r preparing for interviews even working devops ppl are also learning from ur videos that much of relevance is there in ur teaching...i have one question...what is the default behaviour of terraform when we dont create dynamodb table and state lock? how the situation is different before applying dynamodb state lock configuration and aftermath of it?

[@handsomevillian]

Hello Sir, Can we configure NFS mount as a remote backend?

[@faizakashaf805]

thanks abhishek

[@sasmitamallia6960]

what to do if we have to download the Huge size State file that is present in S3 or Azure Storage Account ? apart from downloading manually from bucket ? Do we have any other options?

[@parsiaakash3126]

Hello Bro, At @41:25, when I run terraform apply, it is throwing an error saying that the s3 bucket is already owned by you, I agree that we have created the s3 bucket and then deleted the state file but how is it working fine for you ?

[@gajananwali9705]

Did hands-on on this. And at last tried deleting all resources. But it is showing as destroy complete. But I can see ec2 instance and S3 bucket still running. May I know why

[@AbhishekVeeramalla]

It should not be the case. Please check your state file. Everything thats in statefile will be deleted

[@durgaganesh311]

Hello abhishek ! As we are deleting the state file on the first apply then how we can destroy the created s3 and dynamodb without statefile? and the second doubt is there are two instances are creating rather than one ...can you please explain ?😔

[@nick-sx2zn]

is there any mechanism to execute the resources and files order wise, instead of deleting the state file multiple times

[@saurabhraut3921]

Even if we are storing the terraform state file in the S3 bucket, the drawback of not running terraform apply and then pushing the updated code to repo is still there, right?

[@vinaykpise7998]

Question: I have already provisioned the ec2 resource using terraform later i will update tf file with user data, will this change in ec2 instance.

[@MyGiri123]

Can you please explain what happens internally when we issue terraform init command ?

[@AshokSharma-xv2ks]

To put the state file in s3 we need to first delete the state file we can't able to do in first time ?

[@SriGowthamNallasamy]

Hi Abhishek - If we are using the remote back-end also the terraform show command reveals the contents of the state file . How can we secure it in this scenario

[@AbhishekVeeramalla]

Thats where your IAM policy comes in to picture. You can also use S3 bucket policies

[@SriGowthamNallasamy]

@@AbhishekVeeramalla Thank You Abhishek ❤️🙏

[@InderjeetYadav-f1s]

@abhishek, for eg, we created multiple resources using terraform like multiple ec2, vc, sg. I we can delete a specific resource using terraform destroy command? if we run terraform destroy will it destroy everything which was created previously ? Please help me to understand

[@karthikkumar4233]

Hi Anna, for dynamodb_table why did you use this step--> billing_mode = “PAY PER REQUEST”? If it is mandatory step how does this help in cost optimization?

[@prasadreddy8360]

Super sir

[@AbhishekVeeramalla]

Thanks

[@ChifoAndrew]

Hello Abhishek, pls how can the lock policy be updated? Is it done on the AWS console or can it be provisioned through terraform?

[@devopsTox-bb2cr]

Hi, As the state file contains sensitive data and it will be in s3. What about the file providers.tf which contains aws_secret and aws_key ? How can I secure it ? Can I create it and after the first terraform apply command, delete it ?

[@brownpanther5280]

When we moved state file to S3 and linked locked file to Dyanamo DB , my assumption was .terraform.lock.hcl shouldn't be created in local as locking is taken care by Dynamodb (Like the way i dont see state file ln local ) . However , I do see lock file in local . I created new folder and tried doing same , still lock file is getting created in local . -- Can you help understand this ?Im i missing something?

[@navneetb1231]

When i used terraform with bitbucket, when i pushed my branch to remote repo then i didnt need to separately edit and push the statefile…. Am i missing something?

[@yoshitamahajan353]

Even if we store in git hub repo and forget to push statefile.. cant we always fire terraform refresh command once before starting any changes.. then there wont be any negative side of using statefile in git? Am i right?

[@AbhishekVeeramalla]

How about the sensitive information in the statefile ?

[@ajitsahoo4875]

Hi Abhishek thanks for the lesson. Also we can remove the lock file forcible in some scenarios. Right?

[@charansai9891]

Hi Abhishek. Initially you have created S3 bucket and later we deleted statefile. then again we tried to create S3 bucket with same name and configuration with out state file. and we used remote backend to store state file in S3. But if we use same name and configuration which already used we'll get error right. stating Resource already exists. Can you explain on this.

[@m2abhishek836]

That's correct, he also got the error. See 41:58. But he immediately "clear"ed the screen to escape it. Sometimes these things happens.😀

[@shadowlors13]

Yes, he skiped it. instead of deleting the state file we could have just kept it in the local and use git ignore or something like that to not push the state file to git or any repository and from backend push the state file to s3 bucket. this way it would just push the backend file to s3 and we will not get the s3 error and no extra instance will be created.

[@faisalraj6654]

Hi Abhishek, in timestamp@42:00, when you say, no one will have access to state file/backend.tf, but when devOps is creating a resource, wouldn't the state file required and visible at the root of the repository? Terraform must know the presence of the state file to avoid duplication of the resource. Correct?

[@nehaverma-xn3ed]

Hi, Great video and explanation, one quick question is it good to have aws s3 configuration and resource information in same repo ?? beacause while destroying it was giving error and it should not delete s3 bucket as it consist state file.

[@singhbrozz1996]

awesome explainnation 😊 . can we store state file in database?

[@praveenpundrothu0312]

@Abhishek Even if backend is access to s3 bucket is blocked, "terraform show" cmd will still show the information from s3 to others , right ?

[@kodjoviamedimele3066]

First to comment. Thanks sir

[@AbhishekVeeramalla]

Welcome

[@veeru2310]

🎉🎉🎉wow

[@AbhishekVeeramalla]

Thanks

[@shaikasif4093]

Hi Abhishek Anna. What happened in remote backend and Locking mechanism. If some one pull the terraform config file and made some changes. And without terraform apply he was push the updated terraform config file to Github then how tfstate file will track the changes.

[@AbhishekVeeramalla]

It will not .. thats why we use remote backend. Explained in the video , pls check

[@Jay-yu4yv]

As the other engineers make new changes, statefile will keep on updating. We can enable s3 versioning and can revert back if someone has made some incorrect configurations with the infra. Am I right @Abhishek.Veeramalla??

[@rushilaughingvirus5415]

simplified like a for apple b for ball

[@kittubhai4799]

Hai Abhishek small question For ami id and subnet id we have see aws UI only, without opening can't we do ??

[@ashwinreddy2866]

Hi Abhishek, I have doubt at 20:54 , What if DevOps engineer pushes the Terraform code with out executing it.? He updated the infrastructure but didn't execute "Terraform apply" and pushed it to repo. what happens in that case? Does it lies in the hands of PR reviewer to check ?

[@ravindraravi7435]

Thank you ^^

[@naresh.c2677]

@Abhishek if start state file is in terraform vault is it secure ?

[@manid2900]

Hi Anna...today are you posting the day-5 vedio?

[@AbhishekVeeramalla]

Its already uploaded

[@pavankumardasari6606]

One doubt, We are using S3 to store state file and dynamo db to maintain locking, Does state file also will be stored in Dynamo db or we are using dynamo db only for locking. Just we clarification on dynamo db some more.

[@AbhishekVeeramalla]

Only locking

[@pavankumardasari6606]

@@AbhishekVeeramalla Got it, Thanks

[@aayushisoni3436]

Once you told in this video that terraform.tfstate file is only created when we paas terrraform apply command then after some time when tou ran terraform plan then also terraform.tfstate created. Xan u explain plz.

[@AbhishekVeeramalla]

It is an empty statefile

[@aayushisoni3436]

​@@AbhishekVeeramallayou mean so only during apply this tfstate file generated else it generates a empty file??

[@PremKumar-co5ym]

Hi @ Abhishek sir At end of the series plz give some interview questions and answers.

[@AbhishekVeeramalla]

sure