Kubernetes Policy Management Tools Compared - OPA with Gatekeeper vs. Kyverno

[@Peter1215]

Very interesting comparison, I fully subscribe under the motto of "best tool for the job". If I'm in Kubernetes world, I will tend to choose Kubernetes native tools, so here Kyverno seems like a more fitting choice. Additionally, what I like about any tool is; simplicity and a "lightweight factor". The simpler and more lightweight the tool is, it is generally easier to replace it, easier to reason about and easier to learn. Kyverno seems simpler and lightweight and this is what I like about it.

[@adamsandor1637]

Thanks for making this video Viktor! A couple of thoughts where I think you weren't fair to GK: 1. You are comparing completely different policies - with GK the policy is about ratios of limits and requests, which requires a lot of Rego to define, while the Kyverno policy just says there should be limits and requests. That would be 3 lines of Rego to write. 2. Kyverno reporting policy violations on the Deployment because the Pods that it will create down the line looks fancy but I'm not sure that's true. Judging from the video it looks to me like the Kyverno policy is targeting not a Pod but the PodTemplate part of the Deployment. You can do the same thing with GK and would get the same result. 3. I think it would have been interesting to look at the limitations of Kyverno's language because they are significant both in terms of things you can't do and in terms of ending up with an unholy mess of yaml indentation and dashes. Also no way to separate policy from parameters might make a hello world go faster but what happens when you want to have a bunch of similar but not the same policies? Will you start generating them using templating?

[@DevOpsToolkit]

1. That's my bad. That part was not supposed to compare it with Kyverno YAML, but only to show that Rego can get big and (semi) complex. At the same time, Kyverno does not allow us to do complex things. I made a mistake of not being clear on that one. 2. Kyverno is targeting Pods. You can see that from the `spec.rules.match.resources.kinds` section inkyverno.io/policies/best-practices/require_pod_requests_limits/require_pod_requests_limits/. Now, what Kyverno does below the hood is a different question. It is indeed checking pod templates of the resources that might create pods. 3. Kyverno is more limited than OPA/Gatekeeper. I thought I made that clear in the video, but I obviously failed. My bad. As for the "unholy mess of YAML"... It's Kubernetes. YAML is unavoidable, one way or another. I could agree that YAML might be a mess, that templating might not be ideal, etc. However, that is not specific to policies, but is (or isn't) a general problem in Kubernetes. Using Rego has advantages, but avoiding "unholy mess of yaml" is not one of them. Using Gatekeeper will not result in not using YAML or templating at all, but not using YAML and templating only for a fraction of what we do with k8s. The discussion about Rego is, to me, similar, to the discussion about using Cue in tools like, for example KubeVela/OAM. Is it better than Kube, Helm, or whichever other options is offters? Yes it is. Is it worth learning yet another language? Yes it is, for some. Is it going to become mainstream and used by majority? That's the part I'm not sure. Should we use Rego and Cue and Go template and ... all at once? Probably not. Which ones will survive? I have no idea.

[@adamsandor1637]

​@@DevOpsToolkit Got ya! As for 2 that's quite interesting then if Kyverno can deduce that that Deployment will produce Pods that don't match the policy. For 3. you were very clear on the fact that Kyverno is more limited, but I felt that in this case the devil is in the details of how limited is it exactly. I'll look into that myself a bit more. And you're right, everything in K8s land is a lot of yaml, so it's is quite natural to express policies in this way too. I just feel that policies are somewhat different. You are not just listing a ton of config parameters like with other K8s objects but practically defining a yaml-based programming language. But you're right if that get's you 80% of the way that's pretty good.

[@cajgazachar]

@@adamsandor1637 I do not agree with the 80% part. That is exactly the reason why I did go with Gatekeeper. I do not want to run into something that I want and cannot implement with Kyverno. Would be nice to see what kind of policies cannot be implemented with kyverno that is used normally in prod clusters.

[@chipzoller]

Hi there, Kyverno contributor here. Just a couple of clarifications and relevant additions. 2. Kyverno intelligently translates rules for Pods to higher-level controllers like Deployments, DaemonSets, etc. This feature is called the auto-gen controller (kyverno.io/docs/writing-policies/autogen/). So in Victor's example, yes it really is "that fancy" but also that simple. There's no need (in almost all cases, anyway) to duplicate a policy for Pods N number of times to cover all the controllers capable of creating Pods. Kyverno will do that for you automatically. 3. Anytime a tool doesn't expose a programming language, its capabilities are naturally going to be more limited than another which does. But the matter shouldn't be an academic or theoretical discussion as far as which one might be more limited but rather answers the question *does it solve YOUR problems*? In my experience, seeing and hearing what problems people are trying to solve with a policy engine, anecdotally Kyverno solves more than 90% of those use cases. And that gap narrows even further which each release. Also, as for the ability to not have an "unholy mess" of YAML by using the same policy for different needs with templating, Kyverno has several options including sourcing of values from a ConfigMap, for example.

[@adamsandor1637]

@@chipzoller Thanks for the clarification Chip! The auto-gen controller feature looks really cool.

[@rdr1712]

If anyone needs power & granularity, I suggest OPA version with kube-mgmt which completely removes CRD complexities but still REGO expertise is must. You should do comparison between OPA with side-car, Gatekeeper-OPA & Kyverno

[@samgoldmann1705]

This was an great comparison of these two policy engines. A very thoughtful and fair evaluation of each. This was very helpful to me. Thanks Viktor!

[@isaacegglestone5526]

Awesome! Seems like a no brainer as this is much simpler to implement.

[@mu.abdelhamid]

Thank you for the awesome video - as you usual :D - I think the next ChatGPT should learn from you. I will certainly choose Kyverno for my use case based on this video. 👍

[@manaspecenek]

As time flows, you get younger and younger. Who said DevOps makes people old :)

[@fanemanelistu9235]

Great video, as usual.

[@dudicoco]

From my experience with Rego, it's a pretty annoying language to learn with very unclear documentation. However, one big advantage of OPA as you've said in the end of the video is that policies can be used outside the cluster as well, so you can use the same policies to validate your k8s manifests within the CI when a PR is opened. This is a huge advantage as it saves the time and frustration of having failed deployments due to policy violations. So far I have only used OPA with Conftest and did not try either Gatekeeper or Kyverno, I do wonder if the webhooks policies work with dry-run mode (`kubectl --dry-run/helm --dry-run`)? This would allow validating the manifests within the CI via a dry run and thus make both tools equal in regards of pre-deploy CI policy validations.

[@DevOpsToolkit]

I haven't tried it with dryrun so I'm not sure. I tend to create preview envs for each PR and do the same process as when going to permanent envs.

[@Artur_Martins]

Despite it's almost a 3 years old video, I find the comparison very interesting, @DevOpsToolkit. I would be very interested to see an updated version :)

[@DevOpsToolkit]

I feel that kyverno grew in the meantime abd that if i would do a comparison it would be a very short one: "manage stuff with kubernetes, use kyverno for policies. There is no alternative."

[@SumanChakraborty0]

Great sum-up on the two policy management tool !! OPA is a great product, but the need to learn an extra language "Rego" to operate the policies makes life bit difficult for end-users, where as Kyverno understands Kubernetes better in terms of policy implementation and can be easily implemented through a bunch of yamls :) It would be interesting to see how both the products mature in future since PSP would be removed in Kubernetes release v1.25

[@DevOpsToolkit]

Both products are in early stages and they will likely be very different a year from now. Also, anything related to security (directly or indirectly) is hot and receives a lot of investment money. That means that we will likely have a bunch of other similar tools not long from now. As for PSP... It's dead, so the tools like GK and Kyverno are even more important than before.

[@javisartdesign]

thanks man! really interesting points!

[@bmitch3020]

Seems like Gatekeeper will be the clear winner in the K8s ecosystem. Since k8s itself was the extendable solution that could handle that last 5% of use cases at the cost of lots of extra complexity.

[@hennes131]

Nice video, was waiting patiently since you mentioned it to be released on Thursday. For now, when looking at both tools, I generally got the same impression as. Also you brought my attention now to this dev feedback issue that OPA has. Do you think the „hidden“ violation for the pod could have been more obvious with another CRD that targets pods definition inside the deployment? Lastly I thought gatekeeper is only validating Webhook for now and mutation is not possible at all. However it sounded like it can be done?

[@DevOpsToolkit]

You can solve the "hidden" validation issue by creating a separate CRD for Deployments or, even better, for all the resources that use `podTemplate` (e.g., Deployment, StatefulSet, etc.). Mutation is possible with Gatekeeper, but is in very early stages.

[@leonchik1976]

Thank you for video

[@TheAroraPrince]

Thanks for really nice video. Short question, can we use both at same time ? We already have Gatekepper in place in our cluster and missing some features from Kyverno. So we are thinking to also use Kyverno as well along with Gatekepper . What you recommend?

[@DevOpsToolkit]

I do not see a reason why not to use both if there is a good use case for that.

[@VictorYami]

I love your videos so so much

[@wollginator]

Great, thanks a lot!!

[@chrisjaimon9137]

thanks, awesome video

[@bennewton]

great review

[@riteshpatel8489]

Nice comparison! Currently, policy reports display violations when you deploy policies in 'audit' mode. I believe work is in progress to enable them in 'enforce' mode as well.

[@cajgazachar]

Yes, and you can hook it up into a "report" easily with prometheus+grafana that you use anyway in your cluster :)

[@dharmeshp4697]

Hi Viktor, can you make video on open unison for namespace management in kubernetes?

[@DevOpsToolkit]

I haven't used that one (yet). Do you have a link? P.S. KZbin tends to remove comments with links. Post it without

[@dharmeshp4697]

@@DevOpsToolkit www.tremolosecurity.com/technology/kubernetes and github.com/OpenUnison/openunison-k8s-login-activedirectory ! Please share your views and simple video on setting up open unison for k8s namespace management..

[@DevOpsToolkit]

@@dharmeshp4697 Looks intersting... Adding it to my TODO list... :)

[@조동현-z1n]

Can I use kyverno and gatekeeper together? and there are advanteges when use them together?

[@DevOpsToolkit]

Technically, you can use both in parallel. Practically, I think that would be an overkill with no tangible benefits.

[@chandup]

By the way, congratulations on taking offer at Crossplane. Good choice.

[@DevOpsToolkit]

Thanks. I'm very excited about being a part of a product I believe has a lot of potentials.

[@Peter1215]

@@DevOpsToolkit Congrats, looking forward to in depth videos about all Crossplane products :)

[@DevOpsToolkit]

Just finished another one about crossplane. It'll be published next Thursday.

[@shuc1935]

Can an enterprise decide to keep both for building and maintaining policies of varied complexities? Can they co-exist and allow flexibility to an enterprise? Sorry for the dumb question

[@DevOpsToolkit]

It definitely can and, in general, should. Traditionally, enterprises preferred one tool that does it all. Since then, we all learned that such solutions tend to be bad at everything, instead of being good at something. So, yes. Tools can be combined in general, and in the case of Kyverno and OPA, I do not see the reason why that would not be the case. Now, the more tools you have, the more there is to learn, so it's always about finding the right balance. If in your case, there is a use case for using both, I would say go for it. In the worst-case scenario, you'll learn in more depth how both of those work and will be able to make an educated decision on what to do next (e.g., keep both, keep only one of those, etc.).

[@shuc1935]

@@DevOpsToolkit ❤

[@quant-daddy]

love it :) I'm not using either yet but once I do, this video will play a key role! Would love to know your opinion on kubernetes logging and monitoring solutions. Open source options vs managed like DataDog? I'm using GKE logging and I think it sucks :)

[@DevOpsToolkit]

I almost always prefer SaaS solutions in general, logging and monitoring included. The less things I need to worry about, the better (as long as the price is right). From that perspective, SaaS offers like datadog are great. Now, if you prefer self-hosting it, Loki for logging and Prometheus for metrics are my choices.

[@quant-daddy]

​@@DevOpsToolkit I tried loki and it still seems to take a lot of work in terms of propagating labels and setting up and it's also memory heavy so I agree that Sass solutions are better. Prometheus setup was easy though. Datadog seems very expensive but people seem to love it. Do you have experience using any other SaaS solution for logging and monitoring? I might just go for Datadog. Thank you :)

[@DevOpsToolkit]

@@quant-daddy Whether Loki uses a lot of memory depends with what you compare it. For example, it is lightweight compared to ElasticSearch.

[@mtik000]

I'm also on the DataDog train. Yes, it's expensive, but the "ease of use" factor and features are well worth it, _as long as you can pay for it_. I've never used Loki, to be fair.

[@DevOpsToolkit]

If you can afford it, SaaS almost always beats self-managed solutions. I would not switch from DataDog to Loki unless you have a mission to remove the cost of DD.

[@Babbili]

just for the light mode in Kyverno i'll stay away from it

[@JaydeepDave12]

Gatekeeper is kind of monolith. Kyverno is the future.

[@localho]

Learning a language for policies is going backwards.

[@DevOpsToolkit]

I would not say that it's going backwards. There are advantages to using Rego. The question is more whether those advantages justify it? For some, the answer is yes, and for others it's no.

[@WildLifeBackyardCamera]

very biased comparaison but its probably as you said, coz you didn't understood much about gatekeeper due to its apparent complexity