AWS EKS & Secrets Manager (File & Env | Kubernetes | Secrets Store CSI Driver | K8s)

[@AntonPutra]

🔴 - To support my channel, I’d like to offer Mentorship/On-the-Job Support/Consulting - [e-mail] 👉 [UPDATED] AWS EKS Kubernetes Tutorial [NEW]: kzbin.info/aero/PLiMWaCMwGJXnKY6XmeifEpjIfkWRo9v2l&si=wc6LIC5V2tD-Tzwl

[@AntonPutra]

🟢 [New] Terragrunt Tutorial: Create VPC, EKS from Scratch! (Step-by-Step) - kzbin.info/www/bejne/r5XYeZSFn5iLg8k

[@AntonPutra]

Get Full-Length High-Quality DevOps Tutorials for Free - Subscribe Now! - kzbin.info

[@AntonPutra]

👉 How to Manage Secrets in Terraform - kzbin.info/www/bejne/aX-TpXqBrNt1mqM 👉 Terraform Tips & Tricks - kzbin.info/www/bejne/bYScZaKLid5lsJY 👉 ArgoCD Tutorial - kzbin.info/www/bejne/sHjRlZqafMZkisU

[@AntonPutra]

⏱️TIMESTAMPS⏱️ 0:00 Intro 0:43 Create IAM User with Full Access 1:20 Create Secret in AWS Secrets Manager 2:32 Create EKS Cluster Using eksctl 3:59 Create IAM OIDC Provider for EKS 4:58 Create IAM Policy to Read Secrets 5:42 Create IAM Role for a Kubernetes Service Account 6:14 Associate an IAM Role with Kubernetes Service Account 7:51 Install the Kubernetes Secrets Store CSI Driver 10:51 Install AWS Secrets & Configuration Provider (ASCP) 12:20 Create Secret Provider Class 13:07 Demo

[@AntonPutra]

🔴UPDATED🔴: How to Create EKS Cluster Using Terraform - kzbin.info/www/bejne/g4vco6uwl5ysd7c

[@AntonPutra]

Available on December 27

[@DevOpsEd]

You're a life-saver Anton.

[@AntonPutra]

Thanks Raza!

[@timam7580]

Thanks a lot for step by step implementation tutorial. I manage to mount secret as environment variable in container. But, if I update secret value in secret manager; updated value doesn't show on container.

[@AntonPutra]

There is an additional setting that you need to enable for auto sync. I don't remember exactly which one, but you should be able to find in official docs.

[@franklinpashok]

Hi timam, Can please help me explain how you managed to expose the secrets as env variables? I managed to mount and in the path i can see the values from secret manager but env doesn't showthe values. Do we need to create the secret also to expose as environment variables.

[@franklinpashok]

it's working now, I missed this step to set the two parameters value to true when running the helm chart. enableSecretRotation=true syncSecret.enabled=true. In this video also i missed the section where Anton explains about the cluster role permission to sync with kubernetes secrets. Thanks for the video Anton

[@sharpsaran]

Thank you Very much for the this Video. It really saved my day. I was trying to set environment variable from the Secrets but it was not going well. Your hint in the ClusterRole really helped to solve the problem. Much Appreciated.

[@AntonPutra]

Glad that it helped, were you able to find a way to expose secret value as an environment variable and not as a json object?

[@MohitBishesh]

thanks for the video, just one question, at the time stamp 13:55 for the env: block, you have not mentioned the name of any secret, so what if i have multiple secrets in my secret manager, so which secret will be passed as env?

[@AntonPutra]

welcome, I didn't get a question. Here is the secret name - github.com/antonputra/tutorials/blob/main/lessons/079/nginx/3-deployment.yaml#L31

[@saqlainmushtaq3101]

Great Video You are doing great Job, my Question is it is giving json out put (Key/Value) instead of Environment Variable (Value) which is expected, how can I resolve this for my Application? kindly suggest thanks

[@AntonPutra]

I know it's not very convenient :) At the time of creating that tutorial, only Azure supported key/value. It's possible that they will improve it soon; as a workaround, you can use init container, fetch the json secret, parse and provide as env for the main container.

[@DilanRiz]

Thank you very much for this video, you just explained a lot to me! Good Day

[@AntonPutra]

Thanks Dilan!

[@EquipoSCDF]

Hello Anton! GREAT WORK, one question! What if I only want to create de Secret with the values from the Secret Manager secret, without having to create a deployment?

[@AntonPutra]

Do you mean you want to create "Kubernetes Secret" using values from Secret Manager? Not sure if it's supported, but why would you want to do it? In that way you just add additional object that you need to maintain (rotate/delete etc)..

[@viswaviswa8616]

Thanks for the wonderful video, do we have AWS secret manager and csi driver implementing using terraform including CRD.

[@AntonPutra]

you could use helm with crds or something like kubectl terraform provider to apply the whole folder including crds - github.com/antonputra/tutorials/blob/main/lessons/079/secrets-store-csi-driver/0-secretproviderclasses-crd.yaml

[@yavarkhodadadijokar]

Thanks for the video, is the a way get only the values from secrets? in this video at the end you will get a key-value pair and you have to parse it so you can get the value. I just want to have the values per key and no more parsing.

[@AntonPutra]

I understand, at that time it was impossible. Most likely this feature already implemented.

[@yavarkhodadadijokar]

@@AntonPutra could you please point me to a document on how to do that? I have been stuck on this for longer that I should :)

[@feratimrak7412]

Great explanation. Thank you!!

[@AntonPutra]

Thanks Ferat!

[@regeeshchandran8746]

Thanks a lot for the great video. I was looking for something similar. Just wondering if these works the same way for eks fargate as well ?

[@AntonPutra]

I haven't tried it yet with fargate..

[@joudawad1042]

thank you Anton for the great video, one question related to this approach, do you think that using and mounting values to a volume is a safe approach ? and from your personal experience what do you recommend for working with secrets on AWS EKS ?

[@AntonPutra]

Yes, it's safe mount it as a volume or ENV variable. Just have a rotation mechanism in place.

[@acidbotmaker]

Great video. But I'm curious, if we update the secret value from secret manager will it also effect the secret in k8s?

[@AntonPutra]

It can, there is another setting for that. This tutorial does not cover it.

[@kasavenkateshmpcs9591]

can you please give the helm link to install secrets store csi driver and aws provider

[@harishshankar9055]

Hi Anton, a huge thanks for the great step by step demo, but can you confirm if we can integrate the secrets manager with a k8s cluster running on an on-prem server. You mentioned at 4:10 that we just need to create a service account without creating a OIDC provider ID, but it is failing when we create it.

[@AntonPutra]

Do you mean EKS running on prem, or just k8s?

[@SameerKhan-jv5rb]

please dont put background music it kind of disturbing by the way great video 👍👍👍👍

[@palanisamy-dl9qe]

Thanks for the beautiful video, i have one doubt why we are using CSI driver, can we use FES instead.

[@AntonPutra]

Did you mean EFS driver? I'm not sure if it works with secrets manager or even has a plugin.

[@palanisamy-dl9qe]

@@AntonPutra yes I'm referring EFS driver only.

[@raghavendramerugu]

When I follow the first few steps until the point where you create a namespace and a sa, I see 0 under secrets when I get the service accounts in production namespace, why is it so? Can someone please help

[@caszador]

Hi. At first really great video which is really helpfull. I do have a question which i am unlucky to find any answer to it. It is possible to take values from the SecretManager/Parameter store which are in Key-Value format and store them into a specific file? For example taking values from multiple SecretManager endpoints and mount as single file without using an init/side container to perform it as part of script? I know that it is possible with hashicorp vault just not sure if it is possible with this also. Thanks :)

[@AntonPutra]

To be honest, I don't really remember all the specifics

[@ankitsinha2566]

Thank you for explanation. Is it possible to create a k8s cluster on EC2 instance and then use aws csi driver provider ? Also can we use it with clusters outside aws ?

[@AntonPutra]

Should be posible, instead of creating IAM OIDC Provider just attach IAM policies directly to the nodes.

[@ankitsinha2566]

@@AntonPutra, thanks for replying. I tried to create a policy and attach to Ec2 instance profile, was able to access in with AWS cli but not with provider.

[@stackzombie4046]

instead of passing whole secret object in env. Is it possible to pass each individual secret key with secret value as environment to the deployment ? - name: GOOGLE_CLIENT_ID valueFrom: secretKeyRef: name: api-token like this is what i want to do and pass more like this, thanks

[@a4ldev933]

hmmm.. now this brings up another topic... and that is .. how can assign a kms role to the worker node so that it can encrypt and decrypt encrypted secrets in secrets manager/system parameter store?

[@AntonPutra]

you can use irsa or simply add another policy to the worker node instance profile/role

[@MrVigpotter]

Awesome!! Thank you

[@AntonPutra]

Welcome! :)

[@francistiangco7660]

This is a great step-by-step video, Anton. The github repo is very useful, Thank you! May I just ask, when I update the secret in aws, I can see that it updates the value in /mnt/api-token/secret-token when I restart the pod. However, the environment variable still seems to retain the old value. Would you know if this is expected? And if yes, would you have suggestions on how to keep the secret in-sync? Again, this is a very useful tutorial, thank you.

[@AntonPutra]

Thank you Francis! You can enable auto rotation of mounted contents and synced Kubernetes Secrets by following this tutorial - secrets-store-csi-driver.sigs.k8s.io/topics/secret-auto-rotation.html. I decided not to include this in the lesson since it's still early for it and it's in alpha stage. I wouldn't recommend using in prod.

[@francistiangco7660]

Thanks for that link, Anton! I'll have a read through that. Also, I just realised that you have to delete the secret so it can recreate itself with a new value as you restart the pod. It's just interesting to notice that the mounted secret file doesn't require this step as a pod restart would suffice. Thanks again for this great video - I'll make sure I go through the other helpful videos in your channel!

[@amitpuja100]

Hi Anton, Thank you for the detailed explanation. Could you confirm if Secrets Store CSI Driver works on managed EKS cluster created on fargate as you have created few daemonset yaml files?

[@AntonPutra]

I haven't tried it on fargate, let me see if it works.

[@arindamghosh3299]

Nice informative content. I am trying to create environment variable but it says that the k8s secret is not created in the first place. Though I can view the secret value if I just mount the secret in filesystem.

[@AntonPutra]

Thanks, try to find an error in the contoller log

[@pradippipaliya9675]

I have another question, suppose I want to access the key value directly from secret manager to k8s pod without creating k8s secrets. So that when I run "env" command inside pod at that time the key value is shown. Is it possible without creating k8s secrets?? If possible how to do that??

[@AntonPutra]

At the current stage, it's not possible.

[@piranleon6181]

This is great Anton! I was able to extract multiple secrets stored in Secrets Manager. I have a single query, in case I want to include all the data stored in the Secret created from SecretProviderClass, what annotation should I add in my Deployment? I have tried with envFrom and the name of the secret but it doesn't work. I enter my pod but it does not contain all the data of the Secret, do you have any solution? Thank you very much for the video!

[@AntonPutra]

Welcome, but unfortunately it's been a while since I used it and can help with new annotations

[@alidadgar445]

Hi I had a question about management secret. Thank you for your guidance. All secret management tools need a token or API key to retrieve secrets. Where should we store this token? If we leave token on the server and delete it after getting the information - if the program encounters a problem for any reason and is restarted, we need the token again, but we have deleted it. How to handle the token that is used to get information from the secret manager and keep it safe? Thanks

[@AntonPutra]

in clouds you can use native IAM or IAM for service accounts. With those tokens generated on demand

[@alidadgar445]

@@AntonPutra I am in Iran and do not have access to AWS services. i use akeyless services and get secrets with api in code.What should you do in such cases? ... thanks

[@arpeetaparikh1681]

Liked the secrets manager injection to k8 video but background music is way too loud. I would like to follow and see more if there was low volume or No background music.

[@AntonPutra]

Thanks for feedback, no more music lol

[@joshualegg3750]

Working fine, but doesn't seem to work with secret auto-rotation. What is the best way to apply changes to the secret store.

[@AntonPutra]

Have you tried the latest version of both controllers? Is there a specific error, or did you not find that functionality?

[@joshualegg3750]

@@AntonPutra Making sure the controllers where up-to-date seems to have done the trick. Restarting my pods is now updating the secret. There was no specific error but the new secret wasn't being grabbed. Many thanks, love your videos, always super helpful.

[@AntonPutra]

@@joshualegg3750 I'll release the updated version in about a week. The biggest difference is the ability to parse JSON secrets and mount them as single values, such as 'devops123', instead of {'password': 'devops123'}.

[@soumyakatagi6512]

Is it possible to fetch secret value as an env variable in container instead of a json??

[@AntonPutra]

It should be by this time, pls check docs

[@KishanKushwaha-x4r]

I deployed this with helm but my pods are unable to fetch the secrets from AWS. Getting error : error fetching secrets SyntaxError: Unexpected end of JSON input . Can you please help me ?

[@AntonPutra]

I have updated version of this tutorial and the source code as well, please take a look - kzbin.info/www/bejne/pqGti2ejadljmKs

[@KishanKushwaha-x4r]

@@AntonPutra I did the same thing but still my pod is not able to fetch the secrets as env, it is able to store the secret in specified volume but not working as env to start the application. Any help please ?

[@AntonPutra]

@@KishanKushwaha-x4r check the controller logs, most likely it does not have permissions to create secrets and convert them to envs, this must be enabled - github.com/antonputra/tutorials/blob/main/lessons/196/terraform/21-secrets-store-csi-driver.tf#L11-L12 Also, try to create exactlly the same secrets and test using my examples, after you make it work you can update to your requirements - github.com/antonputra/tutorials/tree/main/lessons/196/12-example

[@a4ldev933]

Very good content... but why going so fast? :-)

[@AntonPutra]

noted

[@aashishnagpal9907]

can you please help iam getting Unable to attach or mount volumes: unmounted volumes=[my-api-token]

[@AntonPutra]

check controller log, you'll get more info to debug

[@aashishnagpal9907]

sorry for being dumb but how can i get controller log because when trying kubectl logs deployment/nginx -n production getting Error from server (BadRequest): container "nginx" in pod "nginx-6b944c497-xx7t8" is waiting to start: ContainerCreating @@AntonPutra

[@aashishnagpal9907]

seems the isuue is when trying to use env variable and for one week i was like

[@AntonPutra]

@@aashishnagpal9907 try checking logs gtom pod from this daemonset - github.com/antonputra/tutorials/blob/main/lessons/079/secrets-store-csi-driver/5-daemonset.yaml

[@piteros]

Problem with AWS Secrets Store driver is that you cannot pass multiple key=value pairs into the k8s secret. What's more, in your example you are passing key=value as an ENV value, not the value itself. To pass only the value you need to create secret in Secret Manager as a plaintext containing the secret value only.

[@AntonPutra]

Thank you! I'll try it out

[@piteros]

@@koushik4531 to be honest, I chose different solution and now I'm using onepassword operator+connect. It works perfectly for me.

[@mohamednehad5871]

great work, just a small note if you accept it, the music is very loud and made me lost focus many times

[@AntonPutra]

Thanks for feedback! I don't use music anymore.

[@mohamednehad5871]

@@AntonPutra would you please share how to encrypt secrets with kms

[@guther]

My God! All this just to access the AWS Secrets?? :O

[@AntonPutra]

yeap

[@AndrewLitvinov]

thank you

[@AntonPutra]

Welcome! I'll release the updated version in about a week. The biggest difference is the ability to parse JSON secrets and mount them as single values, such as 'devops123', instead of {'password': 'devops123'}.

[@mohammedzahranjamali6349]

bro running on 2x speed

[@jaypark2357]

Cool!!

[@AntonPutra]

😊

[@Яслежузатобой-щ7б]

Nihuya neponyatno no ochen interesno

[@AntonPutra]

😊

[@eshgholah]

What kind of audience were in your mind when you created this. This is just going too fast for someone who really wants to learn something. Honestly I have seen your other videos. You need to slow down a little bit and avoid the background music which is used for cooking tutorials. Thanks

[@AntonPutra]

Thanks for the feedback. I don't use background music anymore, and as far as I can tell, my new tutorials are slow-paced.

[@inaslohith3678]

Pls remove background music

[@jitendernegi9074]

L background music.

[@AntonPutra]

thanks i don't use music anymore

[@Botio]

ur cover taste is really bad....god bless u

[@AntonPutra]

Any suggestions?

[@Botio]

@@AntonPutra get a vaction to CA. then ur cover won't be so dark ... even I know u wanna have some hacker style. Or buy some good NFTs, it would help u.

[@pranav2019]

Hi from which location you performed cat new_credentials.csv

[@AntonPutra]

it's been a while, most likely from the lesson dir

[@kkkant1547]

Hello Sir I have one question I have hosted the Docker Image in the ECS My Jenkins server will push the Docker Images into the ECS Now in every 24 hours the ECS Secrete token will expires and I need to add the new Token every time manually Is there is any method in AWS So that My ECS Token will updates automatically inside the Jenkins server so that I don't provide the credentials every-time

[@AntonPutra]

Unfortunately, I don't have a lot of experience with ECS(Elastic Container Service) just yet. I may create some tutorials in the future. Are you referring to the AWS auth token to access AWS services or a token that you store in the AWS secrets manager? If the last, I can think of only doing it manually, you can always write the script to synchronize the secret from AWS secrets manager and Jenkins. Or, perhaps you can retry in the Jenkins, and if you get an error such as permissions denied or token is expired, go and get a new one from Secret Manager.

[@mmukund22]

This works perfectly fine for me when I tried following your solution but I am seeing AccessDeniedException: User: arn:aws:sts:::assumed-role/dev-eks-irsa/secrets-store-csi-driver-provider-aws is not authorized to perform: secretsmanager:GetSecretValue on resource: dev/service/token, For testing I added AministratorAccess permission and it worked but then I manually deleted the POD and it is trying to mount the volume and started seeing that error back. Not sure why?

[@mmukund22]

nevermind I fixed it, Thank you for this demo.

[@mmukund22]

Hi Anton, but I am seeing two problems, 1. secrets are passed in as a JSON inside the containe. Is there any way, if we can export environments individually? right now I am using bash & jq to convert json env variables & parse locally using export but i wish if there's a way we can configure in .yml file. 2. auto-rotations - I know clouple of folks mentioned to use auto rotation but do you have a process or step by step video to do?

[@isurusamarasekara6479]

How did you solve it ?

[@AntonPutra]

🔴UPDATED🔴 How to create EKS Cluster using Terraform MODULES (AWS Load Balancer Controller + Autoscaler + IRSA) - kzbin.info/www/bejne/oYOunpaPeJxkgJY