GitOps Secrets Management with Argo CD

A recording of webinar on Secrets Management with Argo CD delivered by Daniel Hoang, Developer Advocate at Akuity and Nikita Kutselev, Senior Architect.
One of the first challenges you run into when implementing GitOps is how to manage your secrets. In this webinar, we will discuss a few methods for secret management, talk about how they work, and go over the pros and cons of each method.
Then we will show a demo of using external secrets with Argo CD.
GitHub repository: github.com/nikita-akuity/graf...
The know-how from this webinar is also published on Akuity Blog akuity.io/blog/how-to-manage-k...

[@thehyperduck90]

Daniel should definitely start an ASMR channel for Platform Engineering/DevOps. Such a soothing presentation 10/10

[@TechWhale]

Great presentation and demo. Quick question, how do you provide (in the GitOps way ?) the secret needed to configure External Secret so that it can connect to a SecretStore ?

[@rongliao9255]

Great Tutorial - this is super helpful!

[@akuityio]

Glad it was helpful!

[@scottamolinari]

I'm not seeing how sealed secrets won't scale. Could you expand on that please?

[@BretFisher]

I love the idea of Sealed Secrets, but the key for each cluster is different, and I think the default is it roteates every 30 days. So, when you need to deploy that secret to a 2nd cluster, you have to reencrypt and git commit for that cluster. Then, if you need to deploy the same secret to a different resource on a cluster, but it's been 30 days since that last resource was created, then you need to reencrypt the new copy of that same secret, which sometimes you'll only know it's changed the cluster key *after* you have git commit and it failed to deploy. Next, if you delete and try to redeploy a secret 30 days later, it will fail because of the key change, so reencrypt again. Over time, this all ends up with the same secret across multiple namespaces and clusters needing to be encrypted every time, and then reencrypted any time it's redelpoyed or changed. Finally, it's not a "single source of truth" because not everyone has kubectl cluster access, so you'll end up needing to store all those passwords in 1Password or Vault and then copying them into the kubeseal CLI every time you need to apply them in YAML. No one has the decryption key outside the kubectl cluster admin of that key, so there's no central control, auditing, or tracking of where secrets are used. But it's great for demos and mostly great for solo devops people with 2-3 small clusters :)

[@BretFisher]

Oh and another problem I haven't worked out the workflow for. Not every dev has cluster access to use the kubeseal CLI with the public key from the cluster, but they need that public key to encrypt passwords. OK, so just store that public key in the repo where YAML is stored for the resources. But, in 30 days, that public key is rotated and will need to be updated in the git repo, yet there's no tooling to easily know when that key is updated and to auto-PR it into the repo's its used in. Sure, a lot of these rough edges could be solved with tooling, but last I checked those weren't provided with the bitnami-labs/sealed-secrets github project.

[@scottamolinari]

@@BretFisher 1. If you are sharing secrets between clusters, you are doing something wrong to begin with IMHO. If your app needs deployment onto different clusters, that is a different story and the only time I can see scaling issues. So, if you have 100s of clusters all running the same app, that can get messy with sealed secrets, because each cluster will have a different sealed secret. I personally don't have apps running on 100s of clusters and can't imagine I will. So, not a problem for now. 2. Key rotation, as I understand, was a poor naming of the feature. It is more like new key generation. New keys are created every 30 days, and the old keys are still kept "active". So, if you attempt to unseal a sealed secret older than 30 days, it will still work. I'd also say a refreshing of the sealed secret with the newer key can also be easily automated. 3. "this all ends up with the same secret across multiple namespaces and clusters" Same as point 1. You don't create secrets for an app, but rather you create secrets for the app to be able to access other services. If the secrets are being shared between or across security borders (i.e. namespaces and clusters) that is in itself bad practice IMHO. 4. "because not everyone has kubectl cluster access" And they really shouldn't! That is the beauty of sealed secrets, if you ask me. You automate the credential assignment/ secrets creation and basically nobody knows the secret, until it is unsealed and then ONLY those people with kubectl can have access (and even then, on a very much need to know basis).