Day-4 | Terraform State DeepDive | Remote Backend | State Locking

[@madhusaisomarathi]

Hi Bro. Thanks anna for the free courses , i got a good job in one of the MNC company as senior Cloud infrastructure as service role. If every person in IT dedicates some free time for others instead focusing on scraping money from others for time, no one would be jobless. Your initiative is helping others including their dependent families, no other employee would do same as you. Thanks anna.

[@AbhishekVeeramalla]

Hearty Congratulations 🎉 .. Wish you all the best.

[@MHb789]

Did you have any previous experience in IT field?

[@nileshsinha957]

🎯 Key Takeaways for quick navigation: 01:42 📋 *State file in Terraform is where Terraform records the information of the infrastructure it has created.* 03:29 📋 *State file is important because it helps Terraform update and manage existing infrastructure instead of creating new instances or resources.* 07:36 📋 *Advantages of using the state file include updating existing infrastructure and destroying infrastructure in a controlled manner.* 08:05 📋 *One drawback of the state file is that it records sensitive information, such as passwords, by default.* 09:17 📋 *Storing the state file on a personal laptop or any vulnerable device can compromise the security of sensitive information in the state file.* 10:23 🛡️ *Real-world scenarios involve multiple DevOps engineers collaborating on a Terraform project.* 12:12 🛡️ *Storing the entire Terraform logic, including the state file, in a Version Control System can compromise sensitive information.* 14:31 🛡️ *If the state file is not pushed to the Version Control System, Terraform may not know about updates and can delete the corresponding infrastructure.* 16:35 🛡️ *Using a remote backend solves the drawbacks of storing the state file in a Version Control System.* 19:04 🛡️ *With a remote backend, the state file is automatically updated in the external resource (e.g., S3 bucket) when applying Terraform configurations.* 20:41 🛡️ *Storing the state file in a separate remote backend, such as an S3 bucket, enhances security and prevents sensitive information from being exposed in a Version Control System.* 21:09 🛡️ *Multiple remote backends can be used, such as S3 or Azure storage, depending on the cloud provider being used.* 23:13 🛡️ *With a remote backend workflow, DevOps engineers can clone the GitHub repository, make changes locally, verify them with `terraform apply`, and raise a pull request to update the remote state file stored in the S3 bucket.* 27:30 📂 *Before running `terraform init`, there will be no state file available. After running `terraform apply`, the state file will be created and populated with resource information.* 29:03 📂 *The populated state file contains information about the created resources, such as the instance ID, public and private IP addresses.* 30:46 🛡️ *Storing the state file in a Version Control System is not recommended due to security risks and the potential for forgetting to push updates to the state file.* 31:16 🛡️ *Terraform can show the state file using the `terraform show` command.* 31:29 📂 *Deleting the state file can cause Terraform to lose track of the created resources.* 32:10 🛡️ *Using a remote backend like S3 requires creating a `backend.tf` file with the necessary backend configuration.* 33:24 🛡️ *Copying and pasting backend configuration from the HashiCorp documentation or using extensions can simplify the process.* 35:16 🛡️ *S3 bucket creation can be done manually or with Terraform, ensuring unique bucket names.* 37:02 📂 *The S3 bucket name and region are required in the backend configuration.* 38:38 🛡️ *Initializing the project with the backend configuration will create the S3 bucket.* 40:29 📂 *The state file will be created and stored in the S3 bucket instead of locally.* 41:37 📂 *The state file can be stored in an S3 bucket instead of locally.* 42:09 📂 *Using a remote backend allows storing the state file in S3 and accessing it from there.* 42:38 📏 *Modifying the code no longer requires handling the state file manually.* 44:13 🛡️ *Terraform uses locking mechanism to prevent conflicts when multiple people try to update the same project simultaneously.* 45:22 🛡️ *DynamoDB can be used as a locking mechanism for Terraform projects.* 48:15 🛡️ *Adding the DynamoDB lock details to the backend configuration enables Terraform to handle locking.* Made with HARPA AI

[@roshniyadav1539]

Thanks bro it's really helped me a lot I already saw the complete list. just came to revise the concept and your comments save my time.thank u so much

[@EasyTech2000]

Exactly you explain very well like to become zero to hero ☺👍 thanks a lot for such a great inputs and making useful resources... Amazing👍

[@AbhishekVeeramalla]

Thank you 😊

[@ashwinichivate3905]

Gr8 explanation about statefiles and remote backend from end to end.Thanks.

[@AbhishekVeeramalla]

Thanks

[@rehanfazal7354]

thanks GURU-JI for teaching these valuable skills

[@sharan.2822]

those who are using the terraform destroy command to delete all the resources which was created using the following video please make sure that the you need to write the main.tf code in a way that the statefile should be first deleted then only we can delete the bucket otherwise the terraform will throw a error for us saying that the bucket is not empty and yeah this is applicable only for this project only. at last thank you abhishek for making the concept crisp and clear😊

[@AbhishekVeeramalla]

Welcome

[@Y.Haritha-ss5jd]

Thanks a lot bro i spend lot of money but no one explained the way u did..You are a gem..may god bless you bro ❤

[@AbhishekVeeramalla]

😍😍😍

[@anaramurat733]

Hi! Thanks for free but very valueble information. I have had many technical interviews. Trust me, almost in every interview they asked about terraform state file. " What is the best managent policy for statefile?" "How to keep secure terraform state file?' etc.... Greatings from US.

[@deychand11]

Amazon S3, Azure Storage, Google Cloud Storage are the best locations to keep your storage file encrypted. And make sure that you have proper IAM policies in it

[@getgoingwithdeepak]

Hats-off Abhishek for wonderful explaination and practicals.

[@deepufanarmy]

Great session Bayya 👌..Previously I had listened terraform from two resources but still lot of confusion on these state and remote backends ...Now everything is crystal clear ...Thanks Bayya

[@AbhishekVeeramalla]

Welcome 🙏

[@madhugundarapu]

Thank you Abisheek, For spending your valuable time with us. It will really help a lot of people around the globe

[@sainirmalreddygavini2392]

Really ah wonderful explanation Abhishek , Thankyou much!

[@AbhishekVeeramalla]

Most welcome!

[@rakeshkumarnayak8744]

after seen this video i am first time understood the workflow of statefile and his mechanism.

[@rakeshkumarnayak8744]

Thank you sir.

[@AbhishekVeeramalla]

Thank you so much 😀

[@chukwumaonu7687]

Bro, you made this look so easy and very cool for interview preparation. Thank you so much

[@AbhishekVeeramalla]

You are most welcome

[@anilsree06]

Thanks Abhishek!! Able to implement this locking with dynamo db and s3 as remote state backend, It was good hands on practical.

[@AbhishekVeeramalla]

Great 👍

[@ArchuWikky]

Hi Abhishek, At the end of this series, please let us know about the common issues/ frequent issues faced in terraform and interview questions too.

[@AbhishekVeeramalla]

For sure

[@AbhishekVeeramalla]

Btw already I am covering a lot things.

[@ArchuWikky]

@@AbhishekVeeramalla 👍 okay

[@dideepakindorkar3413]

Great sir.. neat and clean explanation.. everyone can understand... Thanks 🙏

[@raghava17]

Hi @ Abhishek sir, Great explanation, Actually im azure guy and i watched all the playlists and i implemented same in Azure Cloud , i got expected Results without any issues while implementing things on azure using terraform backends and modules and with using variables , by saying Thank's is very small word for the wonderful content you delivered 😍✌ 🎉🎉 I'm very happy while implementing same things on Azure by getting expected Results with out Any Errors 🤩🤩🤩,,, Thanks a lot sir 🥳🥳🥳

[@deepufanarmy]

@raghava17 bro do u have any real time azure ci-cd projects ?

[@AbhishekVeeramalla]

Thanks a lot

[@awanishaman1612]

Thankyou so much #Abhishek for this valuable tutorial for free it's very informative and easy to understand for me, very helpful 👏👍

[@msmp4863]

Thank you Abisheek, For spending your valuable time for us. It will really help a lot of peoples in globe

[@AbhishekVeeramalla]

It's my pleasure

[@Pardeep_Sharma999]

Great session ,Thanks Abhishek for your great work

[@AbhishekVeeramalla]

Welcome

[@manjeshsno596]

Hurray, I was waiting for this video and completed with good understanding Sir..thank you so much for your time.🎉

[@AbhishekVeeramalla]

You are most welcome

[@ojosamuel3777]

Great Work @Abhishek. really a refresher course for me. Hope you will share with us the case of using Terraform cloud as remote backend

[@AbhishekVeeramalla]

Thank you so much 😀

[@sandeepsinghrathore505]

@Abhishek.Veeramalla - Super very clear..

[@Shweta.Gaikwad1966]

Thanks for the fantastic video-your clear explanations made everything so easy to understand! 👏

[@gratefultolife2642]

Excellent video so far !!

[@jaswanth_kumar_k]

Thanks for the consistency and efforts ❤

[@AbhishekVeeramalla]

Welcome

[@harshnegi200]

Completed..! Thanks Abhishek.

[@srivanikoppula]

thank uu bro for this seesion very neat explanation and pratical too

[@AtulMisal96]

Hi Abhishek Bhaiya.. At the end of this series please make one video on end to end project on terraform like creation of multiple AWS services...

[@AbhishekVeeramalla]

For sure

[@aishwaryap.s.v.s7387]

hi annaya !!! actually i paid for courses but ur courses are more than those courses .i watched terraform in one stretch and understood the concepts!!! thanks anna for ur help!!!

[@AbhishekVeeramalla]

All the best

[@rekhadevikondeti5764]

Beautifully explained 👏

[@AbhishekVeeramalla]

Thank you 🙂

[@fritzdamiendengambog9169]

Guys do not forget to always hit the like button. And subscribe. Hi bro @AbhishekVeeramalla greetings from Africa Cameroon

[@AbhishekVeeramalla]

😍😍😍

[@agun21st]

I am going to master in Terraform by you sir. 😇

[@AbhishekVeeramalla]

😍

[@hungryman6711]

absolutely a master class ⭐⭐⭐⭐⭐

[@sthitaprangyachhatoi4967]

Hey Abhishek, Please consider creating one Production kind of eks cluster setup using terraform and using all the concepts we are learning in this series.. That would be really helpful to get idea on where to use what i believe.

[@AbhishekVeeramalla]

will try

[@PY-D]

Good idea.

[@adarshjha__1]

Day 4, lesss gooo! 🙌🔥

[@AbhishekVeeramalla]

Thanks

[@mohankumarr1432]

Thank you so much Abhishek ❤😊

[@AbhishekVeeramalla]

Thank you so much 😀

[@mayurichikne526]

Hi sir, one request... By the end of series... Plz make full project of how to create real-time used vpc ,subnet in Aws infra using tf.. Thanks in advance

[@AbhishekVeeramalla]

sure, will try

[@vikramkamble5330]

Best man for devops🙌

[@AbhishekVeeramalla]

😍

[@sadasivagaming]

good explaination

[@nyeemTube11]

Hello Abhishek thanks for the Zero to Hero Terraform Series. Could you please include a few real-life challenges which could be very beneficial for all people who are learning from you.

[@proplusgaming3361]

00:06 Terraform state file and backend 02:43 Using the State file in Terraform 07:36 Advantages and disadvantages of using the state file in Terraform. 10:07 Sensitive information in state files and potential risks 14:59 Using remote backend in terraform to fix the drawbacks of state files 17:19 Store terraform state file in S3 bucket for security and easy access 21:37 Explaining the workflow of updating AWS resources using Terraform in interviews 23:55 Configuring Terraform and AWS in Code Spaces environment 28:12 Executing the terraform apply command creates a state file and updates it with the resources being created. 30:19 Creating a Version Control System-supported project in Terraform 34:35 To create a bucket in AWS S3, you can use Terraform or create it manually. 36:45 Create a unique AWS S3 bucket for Terraform 41:37 Terraform project modified to store state file on S3 bucket 43:48 To avoid conflicts in the terraform project, locking mechanism is important 47:58 Implement backend mechanism with DynamoDB locking. 50:05 This is a simple code that can be executed with the help of documentation and the visual studio extension.

[@mahendrapalla1373]

Hi @abhishek, The way you explaining the content is really good and must appreciate. at the END of this terraform course, are you going to provide video to set-up production ready EKS cluster in AWS for our understandings ? Thank you, Mahendra.

[@AbhishekVeeramalla]

Sure, I will try my best

[@tuonimihart2009]

youre a great teacher

[@AbhishekVeeramalla]

Thanks a lot !!

[@ramthumati2620]

nice brother very understand

[@Jalal921]

Very good explanation Abhi

[@AbhishekVeeramalla]

Thanks a lot

[@shubhamkalekar1519]

Hi abhishek Thanks for your all effort, in real time how the resources will create , whether applying infrastructure using apply command or they use pipeline for infra creation.... If yes then which case we need to create pipeline ams

[@manid2900]

Hi Anna...today are you posting the day-5 vedio?

[@AbhishekVeeramalla]

Its already uploaded

[@ashishpandey8766]

Thanks Abhishek

[@AbhishekVeeramalla]

Most welcome

[@sagarahire-ys3uj]

Amazing content as always...forget ppl who r preparing for interviews even working devops ppl are also learning from ur videos that much of relevance is there in ur teaching...i have one question...what is the default behaviour of terraform when we dont create dynamodb table and state lock? how the situation is different before applying dynamodb state lock configuration and aftermath of it?

[@pritamburnwal6706]

Hi Abhishek, Thanks for the amazing videos of terraform. I am learning a lot with that. at the time stamp of 19:50, the solution for the second issue with local state file I got that, but can you explain a little bit more what exactly the problem is. I got little confused with that issue.

[@cliftonavil]

Excellent 😇

[@itlearntv]

❤ love it

[@AbhishekVeeramalla]

Thanks

[@entertainment_show9597]

Hi Abhishek sir, is it possible to use another database resource for the locking system instead of DynamoDB? Or is DynamoDB the best option for the locking system?

[@ravitejadarla3882]

Hi Abhishek, I request you to make an in-depth video on the Kubernetes cluster step-up on the Ubuntu ec2 Instance there is no proper video on KZbin.

[@Jay-yu4yv]

As the other engineers make new changes, statefile will keep on updating. We can enable s3 versioning and can revert back if someone has made some incorrect configurations with the infra. Am I right @Abhishek.Veeramalla??

[@vikkivikas3685]

don't delete the terraform.tfstate file, it gives bucket already exists, but it working fine. Use terraform init directly without deleting terraform.tfstate file, it will not give any error, I am giving just for information

[@vasu-if3de]

Hi Abhiske, I hvae few doubts on terraform 1. how can we give the IAM policy to the S3 and instances 2. Can you please explain the how can we create the VPC, Subnets with CIDR . CAN YOU MAKE A VIDEO .

[@ManojGandham-dw2xy]

well explained💯

[@kodjoviamedimele3066]

First to comment. Thanks sir

[@AbhishekVeeramalla]

Welcome

[@prasadreddy8360]

Super sir

[@AbhishekVeeramalla]

Thanks

[@sowmithrawulwar2823]

You could've explained this backend concept using modules. Folder structure is confusing and what files to create.

[@ketanmorey1953]

very good series Abhishek sir❤ if we are following this to stored state file in s3 and locked it with dynamodb, if new member of terraform team wants to access state file , then how he access it.. please explain this sir.. Thank you🙏

[@veeru2310]

🎉🎉🎉wow

[@AbhishekVeeramalla]

Thanks

[@umashanker3244]

Hi Abhishek please make interview questions on all devops tools and AWS cloud

[@AbhishekVeeramalla]

Sure

[@umashanker3244]

Hearty Congratulations Abhishek for your newyork poster

[@PremKumar-co5ym]

Hi @ Abhishek sir At end of the series plz give some interview questions and answers.

[@AbhishekVeeramalla]

sure

[@sparun1607]

As suggested by terraform, can we go with option suggested ? will solve if we miss configured ? Initializing the backend... Error: Backend configuration changed A change in the backend configuration has been detected, which may require migrating existing state. 1. If you wish to attempt automatic migration of the state, use "terraform init -migrate -state" 2. If you wish to store the current configuration with no changes to the state, use "terraform init -reconfigure".

[@ajitsahoo4875]

Hi Abhishek thanks for the lesson. Also we can remove the lock file forcible in some scenarios. Right?

[@gajananwali9705]

Did hands-on on this. And at last tried deleting all resources. But it is showing as destroy complete. But I can see ec2 instance and S3 bucket still running. May I know why

[@AbhishekVeeramalla]

It should not be the case. Please check your state file. Everything thats in statefile will be deleted

[@InderjeetYadav-f1s]

@abhishek, for eg, we created multiple resources using terraform like multiple ec2, vc, sg. I we can delete a specific resource using terraform destroy command? if we run terraform destroy will it destroy everything which was created previously ? Please help me to understand

[@MyGiri123]

Can you please explain what happens internally when we issue terraform init command ?

[@nareshb5]

మన వరంగల్ అబ్బాయి యూట్యూబ్ లో క్లౌడ్, devops పైన ఉచితంగా వీడియోలు అందిస్తున్నాడు అని చెప్పా కదా., Topmate అనే సంస్థ ఇతని కంటెంట్ మరియు కష్టాన్ని గుర్తించి న్యూయార్క్ నగరంలో ఉండే టైం స్క్వేర్ బిల్డింగ్ పైన అతని ఫోటో ఇలా present చేసింది.,

[@AbhishekVeeramalla]

Chala thanks andi Naresh Garu

[@pavithrang6760]

Thanks for ur video bro. It helps me a lot to understand about terraform state file. Actually I've one doubt in backend.tf file. When I configure backend in terraform, state file successfully moved to s3 bucket but in terraform file terraform.tfstate.backup file is now created which holds all information about resource. How do I remove this file? Plz provide some tips.

[@singhbrozz1996]

awesome explainnation 😊 . can we store state file in database?

[@ChifoAndrew]

Hello Abhishek, pls how can the lock policy be updated? Is it done on the AWS console or can it be provisioned through terraform?

[@nehaverma-xn3ed]

Hi, Great video and explanation, one quick question is it good to have aws s3 configuration and resource information in same repo ?? beacause while destroying it was giving error and it should not delete s3 bucket as it consist state file.

[@praveenpundrothu0312]

@Abhishek Even if backend is access to s3 bucket is blocked, "terraform show" cmd will still show the information from s3 to others , right ?

[@shaikasif4093]

Hi Abhishek Anna. What happened in remote backend and Locking mechanism. If some one pull the terraform config file and made some changes. And without terraform apply he was push the updated terraform config file to Github then how tfstate file will track the changes.

[@AbhishekVeeramalla]

It will not .. thats why we use remote backend. Explained in the video , pls check

[@FreakinTraveller]

How do we remove lock from remote state file from dynamodb lock table?

[@ravindraravi7435]

Thank you ^^

[@askaralil]

Will terraform show displays sensitive information of state file from s3 bucket of remote backend?

[@pavankumardasari6606]

One doubt, We are using S3 to store state file and dynamo db to maintain locking, Does state file also will be stored in Dynamo db or we are using dynamo db only for locking. Just we clarification on dynamo db some more.

[@AbhishekVeeramalla]

Only locking

[@pavankumardasari6606]

@@AbhishekVeeramalla Got it, Thanks

[@aravindsuresh5848]

@ 42:43 As you mentioned no one can access the statefile but when a user execute the command terraform show, it will show the statefile details so anyway the user can see the statefile details right? Got this question, am I missing anything?

[@feeroz4u]

Question - How does the ".tfstate" file in the S3 bucket get updated in these cases: 1) Another developer raised and got his PR merged without doing 'terraform apply'. 2) Another developer did 'apply' the code, then ran 'terraform destroy', and then got his PR merged.

[@sisindri6981]

1. Another developer raised and got his PR merged without doing 'terraform apply'. As source code merged, if any another resource performs terraform apply command resources will create and update the terraform state 2. Another developer did 'apply' the code, then ran 'terraform destroy', and then got his PR merged. a. if apply terraform apply command resources will create and terrafrom state file will update b. terraform destroy command will delete the created resource and at the same time terraform state file will also update will remove all attributes c. PR got merged but terraform statefile is empty. if some one again run terraform apply it will try to create new resources.. so it is essential to make sure to run terraform plan command before apply to know exactly what going to be happen

[@handsomevillian]

Hello Sir, Can we configure NFS mount as a remote backend?

[@yoshitamahajan353]

Even if we store in git hub repo and forget to push statefile.. cant we always fire terraform refresh command once before starting any changes.. then there wont be any negative side of using statefile in git? Am i right?

[@AbhishekVeeramalla]

How about the sensitive information in the statefile ?

[@BharathKumarReddy2103]

❤❤❤

[@AbhishekVeeramalla]

Thanks

[@charansai9891]

Hi Abhishek. Initially you have created S3 bucket and later we deleted statefile. then again we tried to create S3 bucket with same name and configuration with out state file. and we used remote backend to store state file in S3. But if we use same name and configuration which already used we'll get error right. stating Resource already exists. Can you explain on this.

[@m2abhishek836]

That's correct, he also got the error. See 41:58. But he immediately "clear"ed the screen to escape it. Sometimes these things happens.😀

[@shadowlors13]

Yes, he skiped it. instead of deleting the state file we could have just kept it in the local and use git ignore or something like that to not push the state file to git or any repository and from backend push the state file to s3 bucket. this way it would just push the backend file to s3 and we will not get the s3 error and no extra instance will be created.

[@karthikkumar4233]

Hi Anna, for dynamodb_table why did you use this step--> billing_mode = “PAY PER REQUEST”? If it is mandatory step how does this help in cost optimization?

[@SriGowthamNallasamy]

Hi Abhishek - If we are using the remote back-end also the terraform show command reveals the contents of the state file . How can we secure it in this scenario

[@AbhishekVeeramalla]

Thats where your IAM policy comes in to picture. You can also use S3 bucket policies

[@SriGowthamNallasamy]

@@AbhishekVeeramalla Thank You Abhishek ❤️🙏

[@faisalraj6654]

Hi Abhishek, in timestamp@42:00, when you say, no one will have access to state file/backend.tf, but when devOps is creating a resource, wouldn't the state file required and visible at the root of the repository? Terraform must know the presence of the state file to avoid duplication of the resource. Correct?

[@sasmitamallia6960]

what to do if we have to download the Huge size State file that is present in S3 or Azure Storage Account ? apart from downloading manually from bucket ? Do we have any other options?

[@deepakteja9875]

bro do one video on postgresql state locking

[@Maharaja98Cubic]

Hi Abhishek, your content and method of teaching is good and i personally like that style, so thank you. I have one questions on Remote Backend, what if there are two Devops person access the same file at the same time and during update there could be a chance that second person could miss the update or over ride first persons changes. Is that possible and if YES then what is the solution?

[@AbhishekVeeramalla]

You can implement the locking using state locking. DymamoDB can be used for that

[@sushi9831]

I wish you would have chosen Azure cloud instead. :) Thanks heaps anyway!!

[@AbhishekVeeramalla]

Next time!

[@brownpanther5280]

When we moved state file to S3 and linked locked file to Dyanamo DB , my assumption was .terraform.lock.hcl shouldn't be created in local as locking is taken care by Dynamodb (Like the way i dont see state file ln local ) . However , I do see lock file in local . I created new folder and tried doing same , still lock file is getting created in local . -- Can you help understand this ?Im i missing something?

[@chowdarybvsn]

In real time, are we using terraform for complete setup in AWS? or only for major resources creation only?

[@AbhishekVeeramalla]

Complete

[@durgaganesh311]

Hello abhishek ! As we are deleting the state file on the first apply then how we can destroy the created s3 and dynamodb without statefile? and the second doubt is there are two instances are creating rather than one ...can you please explain ?😔

[@devopsTox-bb2cr]

Hi, As the state file contains sensitive data and it will be in s3. What about the file providers.tf which contains aws_secret and aws_key ? How can I secure it ? Can I create it and after the first terraform apply command, delete it ?