Is CUE The Perfect Language For Kubernetes Manifests (Helm Templates Replacement)?

CUE is becoming the de-facto standard language to define data models and validate data. Should we use it to define Kubernetes manifests? Is it time to replace Helm templating, Kustomize, Carvel YTT, or CDK8S with CUELang?
#cuelang #kubernetes #helm
Consider joining the channel: / devopstoolkit
▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬
➡ Gist with the commands: gist.github.com/vfarcic/a5cb2...
🔗 Cue: cuelang.org
🎬 Helm vs Kustomize - The Fight Between Templating and Patching in Kubernetes: • Helm vs Kustomize - Th...
🎬 Kustomize - How to Simplify Kubernetes Configuration Management: • Kustomize - How to Sim...
🎬 cdk8s - Kubernetes Manifests With GoLang, TypeScript, Python And Java: • cdk8s - Kubernetes Man...
🎬 Carvel ytt Instead Of Helm? A Better Way To Manage Kubernetes Resources?: • Carvel ytt Instead Of ...
▬▬▬▬▬▬ 💰 Sponsoships 💰 ▬▬▬▬▬▬
If you are interested in sponsoring this channel, please use calendly.com/vfarcic/meet to book a timeslot that suits you, and we'll go over the details. Or feel free to contact me over Twitter or LinkedIn (see below).
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬
➡ Twitter: / vfarcic
➡ LinkedIn: / viktorfarcic
▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬
🎤 Podcast: www.devopsparadox.com/
💬 Live streams: / devopsparadox
▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬
00:00 Options to Write Kubernetes Manifests
12:20 What is CUE?
13:38 How CUELang Works and What Does It Do?

[@DevOpsToolkit]

What do you think of CUE? Could it be a replacement for Helm templates, Kustomize, Carvel YTT, or CDK8S?

[@davidbe710]

It is. We started to create our Kubernetes abstractions with it around 4 months ago. It has been a delight so far. Besides everything you mentioned, a big pro is that you actually get meaningful and helpful error message. That's a much better DevEx than what you get with Helm. The schemas are also quite easy to reason about and communicate to devs.

[@cheebadigga4092]

Looks great! One thing that could be better is the schema/struct imports, it speaks "too much Go" to me (or too verbose in that sense). If they can make that somewhat more seamless without using Go CLI directly and without having "verbose Go code" in the .cue file, then I'm all in and will change all my Helm deployments to CUE, no matter the complexity. Thanks for the video!

[@DevOpsToolkit]

@@cheebadigga4092 I would also prefer if imports would not be "too much Go". Still, I don't think that alone is much of an issue. Also, most Kubernetes CRDs are written in Go (not to mention Kubernetes itself), so it makes sense that imports are directly using Go,

[@cheebadigga4092]

@@DevOpsToolkit sure. It's better than nothing for now I guess. The imports thing is by no means a reason to ditch CUE altogether.

[@g00ntar]

I have friend which uses terraform for this purpose.

[@mgwizdala]

Thanks for sharing this. I’ve briefly checked CUE with kubevela, but seems to be nice and valid solution for pain points with current tools. Really looking forward for video with „mystery” tool in action. Cheers

[@Fabian-_-]

I personally use Grafana Tanka and Inka, a CLI similar to Tanka I have created to also write Terraform in Jsonnet and have environments and stuff :)

[@cazador517]

"The more cdk8s code I wrote the more I realize that there are somethings that make my life miserable" I felt that.

[@kdietz65]

It's good Viktor. Thanks.

[@sszynrae]

helm does support schemas via values.schema.json and using them actually make the user experience a lot better, but it relies on chart maintainers to do extra work to generate them. it's not trivial, but the schemas are out there, and schema manipulation libraries exist to at least make it quick to generate a file from something; e.g. a script that imports kubernetes bindings, and mirrors your values.yaml struct interface

[@jamesriordan5461]

I personally use Grafana Tanka for my multi-env helm chart deployment values. I use CUE inside of KubeVela as that is the best for data validation, i.e ensuring the required fields are passed into the KubeVela application types. It would be nice if CUE replaced YAML Helm Chart templating.

[@DevOpsToolkit]

Next week's video will be about replacing Helm with something based on CUE :)

[@TomJordan741]

@@DevOpsToolkit is it next week yet?

[@DevOpsToolkit]

@TomJordan741 almost there...

[@TomJordan741]

I openly wonder if one could use cue as a contained KRM function to leverage it as a generator or validator in a kustomizaton.yaml file. Or even as a transformer.

[@DevOpsToolkit]

@@TomJordan741 I don't see why not.

[@dirien]

Great video as usual! Did you had a chance to play with Timoni (A package manager for Kubernetes, powered by CUE and inspired by Helm) from Stefan Prodan?

[@DevOpsToolkit]

Timoni was the inspiration for this video and will be the subject of the next :)

[@dirien]

@@DevOpsToolkit Awesome! Can’t wait for the next video!

[@arieheinrich3457]

@@DevOpsToolkit try comparing it afterwards with Grafana Tanka

[@DevOpsToolkit]

@@arieheinrich3457 Will do (the comparison). Until then, kzbin.info/www/bejne/Y6LTlKaLjaqfj8U might help.

[@DevOpsToolkit]

It's live: kzbin.info/www/bejne/mJOoYnV8eNhoapo

[@idsvandermolen4549]

I had looked at cuelang last year, but it was missing packaging/libraries/modules/imports . Good thing they added it ! I'm still not very fond of the syntax (if only it would look more like Python/Starlark), but for now it might be the best alternative next to using a generic programming language. Also, I don't want tools like these only for Kubernetes manifests, but for generic Infrastructure as Data and configuration management. So also to manage for example Thanos/Prometheus configurations, or other similar configurations where you need slight variations. Next up is to stick it into a central "stack registry" repo and have pipelines generate all the variants....

[@idsvandermolen4549]

It looks like I'm mistaken. They haven't implemented cue modules/package management yet (i.e. fetch upstream packages from a registry or github). Too bad, this is a very important feature lacking imo for modern language eco systems

[@miran248]

Awesome! Does it support remote modules? Imagine if you could just import the preconfigured 3rd-party service, like with kustomize. Love that it's still close to the original api structure, unlike helm and others. What irks me with helm is the lack of composability - every chart tries to do everything, so there's a lot of repetition between charts, except the api is always slightly different between them, so you have to be very careful when you're integrating them into your stack. With a tool like this one we could create a module, that would handle a single namespace; a module for a single service account; a module for deployment; you get the picture. We'd end up with a standardized set of reusable modules, on top of which we could then build more complex, composable modules. Can't wait to give it a try. (was just asking myself, whether there's anything new in this area, yesterday, and here you are, answering my question :)

[@DevOpsToolkit]

Wait until the next week's video before jumping into it :)

[@oftheriverinthenight]

Im currently using kustomize 5 + helmCharts so I can do overlays on missing values configs. But the main probem I have when deploying is to template the values.yaml itself when I have to deploy helm charts will spend t some time to see if CUE can get helm charts like kustomize and then patch them or operate the generated resources and how it can can be used on argoCD, since CUE looks very interesting

[@anhdeptrai481]

Helmfile supported go templating inside values.yaml as vaules.yaml.gotmpl

[@vanivari359]

Maybe it's because i'm pretty sick right now, but i did not get yet, how this fits into gitops + argocd. Do you push the final result to git? We want to create dynamic environments right now on demand (e.g. INT-2, INT-3...), but that seems painful with Helm and Kustomize because it affects many values in many files (ingress routes, certificates, database-urls etc). No idea we discussed so far looked really great.

[@DevOpsToolkit]

That'll be the topic of the next week's video :)

[@jemag]

Would be interesting to have a comparison of Jsonnet/Tanka vs CUE since they are both JSON superset. I believe CUE is better for validation and Jsonnet is better for templating/reducing boiler plate, however would be curious how it translates in every day use.

[@DevOpsToolkit]

Adding it to my to-do list...

[@renaud72]

Hi, thank you for the video. Its fine for overriding defaults , but what if I need multiple levels of overlays, how can this be achieved ? For example: default -> prd cluster -> standby cluster . Where each level can either use values from upper level or override some of them ? Thanks

[@DevOpsToolkit]

CUE is a language so you can do anything you want like, for example, conditionals based on envs. You can also output to yaml and that means that you can create kustomize resources. Think of it as a programming language that, among other things, outputs yaml.

[@YuriyYarosh]

You can actually define custom validation tags for cdk8s golang and setup any kind of typescript validation to your liking. CUE learning curve is not that great, and other than the learning curve - there's no proper way to describe dependent objects e.g. that RoleBinding depend on Role and specific SA, or some CR's depend on others etc. It's doable with ArgoCD phases/waves, but may end up with undefined intermediate states, in certain conditions. Practically, when we're talking regarding k8s obj management, there's also an aspect of config propagation - e.g. if the target language is suitable for config propagation from 3rd party sources, e.g. if you store a set of variables in a k/v store and can render a ConfigMap or a secret, or propagate as container args etc. Think of it as terraform-consul-sync but crd-k8s-sync, when a CR is changing - all the relevant apps configurations are changing as well. This is achievable with ArgoCD ApplicationSet Go Templates, to a certain degree, but it's not as robust as I'd like it to be (sync delay is too long).

[@DevOpsToolkit]

That will be the subject of the next video when I'll introduce one more tool :)

[@dirien]

@@DevOpsToolkit Whats the new trend of you with all the cliffhangers in every video! 😄

[@DevOpsToolkit]

@@dirien In rare ocasions, I know what will be the next subject, so I can announce it at the end of a video (but without revieling what it is). That does not happen often. In other words, most of the time I don't know what I'll do next so I cannot put a clifhanger.

[@dirien]

@@DevOpsToolkit All good 😂 As I binge watched the last 5 videos of you, I heard you teasing us in the IDP for Infrastructure video and in this video! I love this, builds up the tension for next week! Like a Netflix series! 😅

[@sebastienboulet5728]

Hi Victor, many thanks for your videos, so interesting! @all, I tried to use cue with Argo-Cd as ConfigManagementPlugin, but nothing concluant. Does someone have an hint or a sample?

[@DevOpsToolkit]

I prefer outputting CUE to YAML and storing YAML in Git which is synced with Argo CD. While CUE might be great for those defining manifests, it's not so good for those who just want to see what the desired state is.

[@sebastienboulet5728]

@@DevOpsToolkit ok, thanks

[@zoop2174]

If it can import helm charts & overlay them it'd be awesome. That would help with adjusting things we get from 3rd parties! Right now I'm stuck with using ytt to overlay manifests I template out with a bash script....

[@DevOpsToolkit]

It can't import helm templates. I feel that we are stuck with them for third-party apps forever (but not for our own stuff).

[@danko95bgd]

​@@DevOpsToolkitthat's pretty sad because helm is horrible

[@DevOpsToolkit]

@@danko95bgd I don't mind Helm for third-party apps that much since I only have to "fiddle" with values.yaml file (except when it's not well documented and I have to go through templates). For me, it's much more important how my own apps are defined and my primary motivation is to solve that one with something other than Helm.

[@joebowbeer]

Why is kpt not among the choices?

[@DevOpsToolkit]

You're right. I should have included kpt in the list of choices.

[@gauiis]

I would probably go with cdk8s, then Pulumi or Terraform CDK as it seems to be what the big companies are leaning against. See for example the AWS Constructs project and their CDK. Everyone seems to be aiming for managing infrastructure/manifests with code, not DSLs.

[@DevOpsToolkit]

That depends on what is code. CUE is code but leaning more towards declarative instead of imperative direction. CUE is not a DSL. Also, big companies are leaning first and foremost towards kubernetes and that might lead towards CRDs which lead to CRs which tend to be only a few lines that can be managed with anything.

[@DevOpsToolkit]

Now, behind those CRDs are operators which, indeed should be written in code.

[@IvanRizzante]

I think that tanka is an excellent choice rather than pure jsonnet, kustomize or whatever, even though it doesn't support the schema import requirement you set. In my own experience I didn't find anything so complicated that couldn't be managed by tanka.

[@ehakan99]

Even with great functionality, CUE doesn't have a proper language server / vscode plugin yet. I've definitely missed completions, refactoring, go to definition that come built in with cdk8s + ts.

[@DevOpsToolkit]

Cue plugin for vs code works fine for me.

[@guai9632]

I made a gradle plugin which uses kotlin-dsl generated from k8s/openshift swagger schemas to generate kube yaml files. Idea validates it for you as you write it Nobody uses it though :(

[@DevOpsToolkit]

It's very hard to make projects known and result in usage.

[@c_kemper]

Cliff hanger!😆

[@VishalDongre-pg9un]

What about pkl lang?

[@DevOpsToolkit]

Here it goes: kzbin.info/www/bejne/hJ6UmqKNhbeFjLM

[@VishalDongre-pg9un]

Thanks, could you please make a video on how you learn new technology and tools? I've joined a new company as a DevOps engineer at a startup and I feel overwhelmed by seeing all those tools-Argo, Kustomize, Helm, Istio, etc. It seems like they are using each and every tool from the CNCF Landscape. It would be really helpful for newcomers like me if you could provide a framework for learning new things. By the way, you're awesome! I've learned a lot from you.

[@DevOpsToolkit]

@VishalDongre-pg9un unfortinately, i do not have any special framework for learning. The only thing I do is pick up a new tool, learn what it does, and figure out how it fits into the "big picture". I pick them randomly. Figuring out what some tool does is easy. Understanding how it fits into the system is the hard part. Also, 9 out of five tools are dead ends so much of what i explore ends up in trash and many of those never make it into my channel.

[@sarceviczm]

This seems like a simpler version of jsonnet.

[@DevOpsToolkit]

It is a combination of json and go so there are obviously similarities.