How to apply policies in Kubernetes using Open Policy Agent (OPA) and Gatekeeper

[@DevOpsToolkit]

What did you think about OPA and Gatekeeper?

[@denverfletcher9419]

I don't even trust me to not do something stupid! What's your view on VMWare Cloud Universal?

[@DevOpsToolkit]

unfortunately, I haven't used it (yet) so I can't say much about it :(

[@gvoden]

as pod security policies are supposedly being deprecated this may help with a security baseline for infrastructure teams

[@ThoriumHeavyIndustries]

With PSP gone, I am really interested to see if kyverno or opa/gatekeeper will make it to the default. OPA has the downside of "rego" and people tend to always go for the more convenient or more easy approach. Since opa/gatekeeper is part of Openshift and RedHat is a big driver here, both options seems to be realistic. In regards of on which horse you should bet... so your implementation effort is not lost (again)... we will know more in about 12 month when psp has aged out in k8s 1.25.

[@DevOpsToolkit]

@@ThoriumHeavyIndustries I like Kyverno, but, seeing the adoption of the two, I would place my bets on OPA, especially since there is an increasing number of projects using it.

[@MatiFix]

Greate video! Thanks so much for the time to work this on! Greetings from Argentina!

[@GeraldOSullivan]

Thanks for yet another very informative video. The bowling alley analogy is excellent: go mad and smash your own pins, but don't mess with mine.

[@DevOpsToolkit]

That should be the goal. We should not prevent others from "playing" but makes sure that they do not ruin the "game".

[@randomthoughtstech]

Excellent video and a good analogy to bring it all together. OPA is awesome!

[@superoe]

great introduction to opa !

[@zahurulhaque6377]

wonderful video Viktor, awesome job. I like to see how we can bring the argo in the scene.

[@DevOpsToolkit]

That's a great suggestion. Adding it to my TODO list... :)

[@wollginator]

Thank you for the great video!! You have considered very good examples, I'd be indeed very interested in how this works together with ArgoCD (and potentially the whole Argo CI/CD stack), this is gold!

[@DevOpsToolkit]

Gatekeeper with Argo CD is on my TODO list, but I'm not yet sure when it's turn will come. Your comment bumped it towards the top of the list :)

[@albertolosadagrande3755]

I agree with Roger, in the limit's examples, I think limitRanges fits much better since it is already built-in. I think there is another example in the library forbidding pull images from non-allowed registries, which I think it can be done at runtime level which I feel more native than messing with REGO. But, yes, on the other side, I take it as an example of what you can do with OPA/Gatekeeper. Really great video and explanation Viktor. Gràcies!

[@farhadkazemipour4247]

Awesome video, thank you for giving us a good understanding.

[@ilyasskaouam1896]

Thank you for the great video

[@develom_ai]

Great video!! Thank you :)

[@grizzle8911]

One observation I have only just starting to learn more about OPA/Gatekeeper and admission controllers would be that it would be ideal (IMO) if the policy was able to identify all issues upfront without partially creating your deployment and then having to go dig through the various resources to find out which one it failed on. If my initial kubectl apply failed giving me a list of 6 things I needed to fix I would do them all in one hit rather than fixing one thing and crossing my fingers.

[@taosong4453]

Great post. Is there a way to perform the policy check and get a report at build time at CI/CD before hit the cluster.

[@DevOpsToolkit]

You can set `enforcementAction: dryrun`.

[@nickneek80]

You can run OPA policies using conftest (see github) - so yep, you can do it all local or in CI (before it even hits the cluster).

[@AllForArtYS]

Thanks for the great content!

[@DevOpsToolkit]

My pleasure!

[@CharlesVosloo]

Thanks for the great content! I have a two requests for future videos: (1) How to integrate OPA and gatekeeper with ArgoCD, which now allows for Policy as Code, not possible under the PSP (Pod Security Policy) way of doing things. (2) I gather you are not a fan of managing your own cluster. What about exploring Kubematic (KubeOne)? An open source solution that can automate the operational side of running a k8s cluster (like updates), using kubeadm and works on prem too

[@DevOpsToolkit]

Thanks for the suggestions. Adding both to my TODO list...

[@johannesprinz]

Very cool thank you, is it possible to get the policy violations even earlier than deploy time? Like in the editor when authoring the deployments. Pushing left as far as it goes ;)

[@DevOpsToolkit]

You should be able to use `--dry-run server` arg in `kubectl apply`. As for editors... If your editor is k8s-friendly and supports running kubectl commands with `--dry-run`, it should work with Gatekeeper just as with any other k8s resource.

[@andreykaliazin4852]

Super stuff! Would it be possible to apply OPA on the ops level cluster, so the. rules were enforced on the devs level within an allocated vcluster?

[@DevOpsToolkit]

It depends on OPA rules. Those affecting pods, services, and ingress can be applied on the host cluster since vcluster delegates those to the host. All other policies should be in a vcluster. The good news is that most of the policies apply to pods, services, and ingress.

[@tigercat123456]

Good video & demo of OPA! Can you somehow scope policies to eg namespaces? In the case you want to generally limit CPU resources to 1000m, but "production" namespace is allowed to use 10000m?

[@DevOpsToolkit]

Yes you can. You can see an example of that in the video. The last policy violation was executed only after I switched from the `default` to the `production` Namespace.

[@Chr15Murray]

This does look like a nice tool and something i’ll be exploring - thanks for the video. Was there any reason not to apply the policy to the replicaset instead of the pods? I feel this would give a better user experience. You would still need the pod policy too in case someone tries to create one outside a replicaset

[@DevOpsToolkit]

As you said, you need a Pod policy since Pods are the lowest denominators in k8s. Pods can be created through ResplicaSets which are created through Deployments. Or they can be created through StatefulSet, Knative, and quite a few other resources. On top of that, quite a few tools are creating Pods directly (e.g., Jenkins, Argo Workflows, etc.). So, policies applied to Pods are a must but, as you mentioned, that does not mean that similar (if not the same) policies cannot be created on higher-level constructs. If you do that, you'll make a better user experience.

[@mdjaere]

Great demo 👏 💯 Another could be setup of FluxV2 multi-tenant with OPA

[@DevOpsToolkit]

Adding it to my TODO list... :)

[@TheBestDanceMoves]

Alright thank you. I have a problem. Let me detail my problem. Consider that we have two users/developers, John and James. Now, I have a kubernetes cluster with two pods inside, pod1 and pod2. I want John to access pod1 only and not pod2. similarly, I want James to access pod2 and not pod1. The same scenario occurs if I want them to access nodes. I hope it is clear now. I look forward to your reply, thank you

[@DevOpsToolkit]

Create two Namespaces, one for each of those persons and define RBAC that allows them to access a specific Namespace and perform specific operations in each. Use policies (like Gatekeeper) to define which properties of the resources they are allowed to manage are acceptable. Ideally, don't allow anyone access to anything directly. Use Argo CD or Flux instead.

[@TheBestDanceMoves]

Alright thank you for this. But to be honest I am a little bit lost and if you could do a video demonstrating all this in details, it will be great outlining each step clearly and testing. I will really appreciate @@DevOpsToolkit

[@DevOpsToolkit]

@TheBestDanceMoves I just added RBAC to my to-do list :) Cant promise the date just yet.

[@TheBestDanceMoves]

Thank you, I look forward to it

[@joebowbeer]

Comparison of Gatekeeper vs Kyverno? Kyverno is more comprehensible?

[@DevOpsToolkit]

I have it on my TODO list and I hope to make it soon.

[@DevOpsToolkit]

I just published a video about Kyverno (kzbin.info/www/bejne/eoOom62cid-BpqM) so that both are equally represented. The comparison between the two will follow (hopefully soon).

[@mohamedsamet4539]

For "Require resource limit policy" and for "Disallow latest tag policy", it allows creating the deployment then it blocks creating pods. It should prevent creating anything that does not meet the policy.

[@DevOpsToolkit]

Exactly. Still, I'd prefer if it would be a bit more "intelligent" and block creation of parent resources (e.g., Deployment) or, at least, provide a direct response indicating that Pods will not be created. This is especially important in automation. I could have a pipeline that finishes executing successfully and be oblivious that it is actually blocked. That specific feature is better solved in Kyverno (kzbin.info/www/bejne/eoOom62cid-BpqM).

[@mohamedsamet4539]

I had already watched your video on keyverno and I like your videos and explanations btw. About keyverno, do you think it is ready to use in big projects? Is it stable/secure enough to use in production?

[@DevOpsToolkit]

@@mohamedsamet4539 I think that the core features (those around policy enforcements and audit) are ready. The project is doing more than that (e.g., mutations, reporting, etc.) and those feature might not be fully there. Nevertheless, those features do not exist in Gatekeeper anyways. What matters is that the "core" policy-related features are working fairly well.

[@joebowbeer]

How easy would it be to create a more _user-friendly_ rule that required that a limit exist *and* that it be less than a specified value?

[@DevOpsToolkit]

That depends on what you mean by user-friendly. If a user is a person that writes the rules, rego is anything but friendly.

[@joebowbeer]

What I'm getting at is that having the rule written in two parts can result in the manifest author needing to fix their manifest twice.

[@DevOpsToolkit]

I think you can write multiple expressions with rego. I haven't tried it myself since that also means more duplications unless you have very uniform type of resources.

[@caesarion6]

OPA seems to overlap a bit with ResourceQuotas and LimitRanges

[@DevOpsToolkit]

That's true. They are similar in the sense that both limit users what they can do. The major difference is that there is a limited number of constraints we can do with "core" k8s. OPA is meant to allow users to define policies for anything in k8s (and beyond).

[@gvoden]

@@DevOpsToolkit we are looking at some commercial products that have an admissions controller, Check Point Dome9 has an Early Availability product...

[@DevOpsToolkit]

I haven't tried Check Point (yet). Will do :)

[@ziaurrehman4738]

after applying the constraints, I am still able to create a node port service. it's not working

[@DevOpsToolkit]

OPA can be tough to debug.

[@ziaurrehman4738]

@@DevOpsToolkit Although I try to create another constraint to disable to create namespace without label, but that constraint work.

[@DevOpsToolkit]

@@ziaurrehman4738 I'd need to check the manifest of what you're using before saying what might be the issue. You can send me a DM on Twitter (@vfarcic) or LinkedIn (www.linkedin.com/in/viktorfarcic/). Also, you might want to try Kyverno as well.

[@ziaurrehman4738]

@@DevOpsToolkit thanks, i sent the invitation on linkedin

[@hackerops1407]

Can you make video on how to check Kubernetes audit logs via Falco

[@DevOpsToolkit]

I can and I will. It's already on my TODO list. I'll "bump it" so that it's closer to the top.

[@abhishekkhanna1349]

how can we remove this template and constaints?

[@DevOpsToolkit]

You can execute the same `kubectl apply` commands but as `kubectl delete` instead. That should remove the templates and constraints.

[@ThoriumHeavyIndustries]

No, it is not trust. It is experience, and on the other side missing knowledge. With DevOps in K8s many things shifted to "Developers", they have no or only very basic understanding of, starting with Networking, Firewalling, OS-Level Administration over container security,, software security to Pod/Project or even Cluster config. 8 out of 10 will use root / cluster admin / privileged containers for everything simply because it is easy, fast and works. That isn´t meant as an insult, but being realistic. Someone screwing together an aircraft, does not mean he can or should fly it. :D

[@DevOpsToolkit]

You're right. Nobody can know everything. That's why we have different types of experts, all working together towards a common goal. That's also why off-the-shelf solutions do not work. That's why ops/SRE/etc are working towards making company-specific services that will do whatever needs to be done by whoever needs to do it, instead of waiting until things are thrown over the wall from one group to another.

[@ThoriumHeavyIndustries]

@@DevOpsToolkit Exactly, and that is why I see shift-left critical. It sounds sexy and it feel good ... fast, agil, less paperwork, communication, meetings and other overhead but more coding, no one watching your fingers, higher "productivity"... and that is where dragons dwell. We see that with public cloud service, people not paid for or interested in "overhead" like security, legal, compliance, safety and continuous operating using the service "which look and feel" very easy - until you are in the 85% of companies with at least once annually compromised cloud workload. - Wide parts of my career I used to be a developer myself, and I can totally understand why and how this happens - even considering the mountain of free and opensource tooling helping you massively today to make it better... when I look back to my time, I had Vi as an IDE and $home was the gitlab and the "development compartment" was a PC build from leftover and scrap parts . :D

[@DevOpsToolkit]

@@ThoriumHeavyIndustries The problem is that you cannot shift-left by saying "now you do it, even though I know you cannot". For that to be successful, shift-left means that people on the right are creating services that people on the left can consume. So, instead of using AWS directly, create a service that will give people freedom to do what they need to do while hiding the complexity they do not care about or know of. Make that service secure. Add company's best practices into it. And so on and so forth. In other words, what I'm trying to say is that people on the right (ops) should enable people on the left (devs) to do stuff not by saying "do it", but by saying "this is the service you can use to do X". What AWS/Azure/Google/Kubernetes is to the people on the right is what those services are to the people on the left. AWS/Azure/Google/Kubernetes are not meant to be used directly. They are building blocks waiting to be assembled and exposed as a set of services.