Multi-Tenant Kubernetes Cluster With Capsule

[@DevOpsToolkit]

How did you implement tenants? What do you think about Capsule?

[@MrTmorton77]

github.com/clastix/capsule-proxy Should address the list namespace issue. Although I am just now digging into all of this. Thanks for all the great content!

[@MrTmorton77]

$ kubectl --context alice-oidc@mycluster get namespaces NAME STATUS AGE gas-marketing Active 2m oil-development Active 2m oil-production Active 2m

[@DevOpsToolkit]

@@MrTmorton77 I think I mentioned it in the video. I don't think the proxy is the solution but, rather, a workaround. That would need to be changed in Kubernetes itself to make it transparent for the users.

[@MrTmorton77]

@@DevOpsToolkit you did I just commented before I heard it. :) Looking forward to your video on HNS.

[@shilashm5691]

Can we use multi tenant concept for maintain different env(dev, stage, prod)?

[@mehdik3883]

Your description of tenant and multi-tenant with apartments example is just awesome ♥️

[@harshthakur7215]

Hey Viktor, would love to hear more about your thoughts on hierarchical-namespaces.

[@DevOpsToolkit]

Adding it to my TODO list... :)

[@kevinyu9934]

I like this type of contents!!! I like innovative Cloud Native Solutions! Keep it up bro

[@Peter1215]

Multitenancy is a very hard problem to crack, capsule looks promising. I would also like to see a deep dive on hierarchical namespaces please.

[@DevOpsToolkit]

HNS is coming... :)

[@javisartdesign]

thanks. This tool seems very promising. As yoou said sometimes kubernetes clusters can be used for prototyping, educational, testing, etc.. and using Capsule can provide this level of isolation to sharre the same infra resroures at the same time

[@DevOpsToolkit]

Capsule is indeed awesome. Nevertheless, I think I have an even better option. If everything goes as planned, it should be published next Monday.

[@TAICHI1SCO]

Hey Viktor, thanks for the video Can you do a video on multi-cluster/tenacy monitoring using using prometheus-stack and thanos/cortex with HA

[@DevOpsToolkit]

That's a good one. Adding it to my TODO list... :)

[@marcin_karwinski]

I wonder if Capsule could be paired with some WebUI/GUI for limited scope of actions but full UI for tenants...

[@DevOpsToolkit]

I'll have a video about that in about 3 weeks :)

[@SumanChakraborty0]

Thanks Victor for the wonderful explanation around multi-tenant clusters. How do you compare Capsule with other CNCF certified projects such as SAP Gardener or even Kubernetes Kubermatic Platform? These are really cool solutions to manage multitenancy across multi-geo locations.

[@DevOpsToolkit]

Capsule is mostly focused on solving multi-tenancy problems within a cluster while those you mentioned are more focused on how to manage multiple clusters. I'm planning to do a video about both. In the meantime, a new video about a potentially better solution for multi-tenancy will be published in a few hours :)

[@cajgazachar]

Hi Viktor, thanks for the video, capsule looks interesting. Would you mind to make a video about automating cluster bootstrapping? Let's say, we provisioned our k8s cluster with a tool like terraform... Then comes all the other components which are required for real world production multi-tenant clusters like: argocd, sealed secrets, gatekeeper, istio, trident, prometheus, fluentbit, RBAC policies, namespaces for projects etc... How would you automate this? What we started to build up is that when we create the cluster the last step in the provisioning with terraform is to setup argocd and use app-to-app pattern to bring up all these components (starting with sealed secrets and a key restore). This seems to be working (we are at very early stage) but we would be interested in your thoughts :)

[@DevOpsToolkit]

I do something similar. I would create a cluster with Terraform and do the initial installation of Argo CD and the root app which, initially, would have only Sealed Secrets and Argo CD (so that it manages itself). From there on, it's all about adding the rest of the resources to Git repos and letting Argo CD do the "magic". That being said, I'm moving towards a different model in which Crossplane would replace both Terraform and the initial k8s resources like, for example, Argo CD. On top of that, I'm using Crossplane Composites to simplify the process in a way that anyone can define anything in a very simple way. I'll be making a video (or a video videos) on that subject soon.

[@stevenb7814]

@@DevOpsToolkit are you saying that Crossplane can replace Argo CD all together or are you just talking about for bootstrapping a cluster? I would love to hear more about the solution you are working on. I love your videos!

[@DevOpsToolkit]

I'm saying that crossplane can replace terraform as well as to do initial installation of Argo CD (even though that part is not important). Since crossplane is based on k8s resources, it is a great combination with Argo CD (not a replacement). With crossplane managing your infra, you can have everything aynced with Argo CD (not only your apps).

[@anshuman2121]

Nice informative video. Does capsule and open policy agent both has same work?

[@DevOpsToolkit]

Conceptually, there is overlap. Part of Capsule is about ensuring that tenant "behave". However, that's only conceptually. In practice, it would be very hard (not to say impossible) to do the same though OPA simply because Kubernetes itself does not have multi-tenancy baked in. If you would use OPA for validating tenants operations, you'd still need a solution for creating tenants.

[@jonassteinberg3779]

nice 1. or should I say, "nyice vun". elite videos!

[@sergiykulanov5837]

Hi Viktor, Thank you for this video. Did you have any chance to check/use kiosk from loft for multi-tenancy in k8s?

[@DevOpsToolkit]

Not yet! Adding it to my TODO list... :)

[@yassinemessaoud8619]

This video came out in the right time thank you !

[@alex24x7]

Hi Victor! Thank you for review. Is it better than Loft?

[@DevOpsToolkit]

It's different. Capsule makes multi-tenancy "invisible" for users, but its scope is smaller than Loft. I'll do my best to create a comparison video.

[@DevOpsToolkit]

Just published a review of Loft vcluster :) kzbin.info/www/bejne/gKKlm6OspZdpbrs

[@alex24x7]

@@DevOpsToolkit Good job, thank you Victor!

[@autohmae]

At least one provider delivers managed k3s, thus having a lot less overhead for the control plane and still having per customer/tenant clusters

[@DevOpsToolkit]

Oh yeah. When using k3s abd with something as fast and cheap as, let's say Civo, it does not make sense to create virtual clusters. However, most of k8s usage is in AWS, GCP, and Azure and they are slow and cost a lot for such usage.

[@autohmae]

@@DevOpsToolkit yes, you (unintentionally ?) guessed right it was Civo which I had in mind. It really makes you think: do we really need k8s for new clusters ? How much am I missing with going with k3s ?

[@DevOpsToolkit]

@@autohmae Let's put it this way... For larger clusters, the difference in the time to create a cluster or the resource overhead is not big (relatively speaking). So, i would not go with k3s for HA cluster with many nodes. It's not worth it. I still prefer GKE, EKS, AKS, etc. But, if you have smaller clusters and, especially if you create/destroy them often, those differences are relevant and using k3s makes a lot of sense. I might, for example, use GKE as the production cluster, but Civo for development or preview clusters that are created/destroyed on-demand and relatively frequently.

[@autohmae]

@@DevOpsToolkit my thinking is where I want to move to is gitops+Cluster-API. So yeah.

[@viveknidhi]

Nice 👍