How to run local multi-node Kubernetes clusters using kind

[@DevOpsToolkit]

I stated that minikube is a single-node cluster and that there cannot run multiple clusters. That's not true. It can have more than one node through the `--nodes` argument, and it can run multiple clusters through its profile. However, while those things are technically true, due to its requirement to run VMs, those things are not working well (at least not when compared to kind), and the level of control is considerably lower. It can provide similar functionality as kind, but it's far from being useful in multi-node multi-cluster scenarios.

[@captainkill8433]

Minikube supports multiple nodes, but currently just one control-plane with several workers. This is a really good video. I learned a lot. Thank you!

[@ofir2565]

Thanks Viktor! Isn’t minikube with Docker as its driver become somewhat equivalent to kind?

[@DevOpsToolkit]

@ofir2565 it indeed is.

[@sekirandahamza1260]

I like how the tutor talks casually. Like he is seeing you and you're in the same room just having a talk.😅 It helps understand the content. Thank you for the video anyway.

[@ColinFox]

There is one difference between Kind and Minikube that FORCED me to not use minikube - and that is that minikube has its own internal docker server SEPARATE from whatever you have in your environment. In my case, minikube was using docker v 20, and my linux box was at 19. There is a conflict between these two versions that would prevent local file/volume sharing, which completely broke our build process. Kind, however, is just running inside whatever docker environment you have - it doesnt' have its own docker server, so there's no conflict. It also seems to be lighter on the system than minikube. I've been very happy with Kind so far!

[@DevOpsToolkit]

Kind is better than minikube. It uses less resources, it is faster, and it is closer to whatever you're running in production. On the other hand, k3s is even better except the last part of being closer to production.

[@quackycoder9565]

Getting myself into Devops, and your video and explanations are amazing! Please keep sharing your knowledge! :)

[@DevOpsToolkit]

Glad you like them!

[@igorfrezer6642]

Thanks! Amazing that I met you today in KZbin and got two helpful tools for my work!

[@Clobercow1]

I just got started using k3s and I like it.

[@Parm_newchannel]

Awesome explanation Thanks you for sharing this information

[@subhasishbehera1712]

Loved your presentation.

[@kulinskyvs]

Thank you! Was interesting to discover kind. Personally, I'm using k3d (which is actually k3s in Docker) for local k8s clusters - it has ingress controller out of the box (Traefik).

[@DevOpsToolkit]

k3d is a good choice. I'm planning to review it in one of the upcoming videos.

[@DevOpsToolkit]

Done. Just released a video about k3d. kzbin.info/www/bejne/o3TIpKh9oJJ5odU

[@m19mesoto]

Hello Victor, Excellent video !!! UPDATE: * In recent time minikube also support docker as driver. But never the less you are right kind seems to be much more lightweight solution. * I am also using kind + podman (mimic docker)

[@DevOpsToolkit]

In the meantime, I switched to Rancher Desktop. That's the only container engine and Kubernetes running on my machines for a while now.

[@m19mesoto]

@@DevOpsToolkit i was using the same with the moby engine on my company mac, however on my personal pc, with Fedora podman is just great..

[@amabamo5769]

i tried all and k3d is a winner for me

[@DevOpsToolkit]

Rancher, in general, is awesome. I should probably make a video for each of their products.

[@DevOpsToolkit]

Done. Just released a video about k3d. kzbin.info/www/bejne/o3TIpKh9oJJ5odU

[@MiguelPintolookatitude]

Great video but you forgot something else you can easily do, you can choose the k8s version that cluster or even node is running. - If you're testing a k8s based app and you want to test on the newer version (or even automate such tests by using the config file you can replicate the cluster each time) - If you wanna make sure your running the same version as the cluster you're deploying into... Just a couple of scenarios where these feature with multiple clusters and being able to choose the k8s version you're running are huge to me.

[@MiguelPintolookatitude]

remembered another one, since you can change the version of k8s at the node level what if you want to test what happens to your apps in the middle of a k8s version rollimg update where at times the nodes of you clusters are running different versions for a time... :)

[@johngrabner]

Wow, very detailed and easy to understand. Would love to see kind vs microk8s

[@DevOpsToolkit]

Adding it to my TODO list... :)

[@akashdeshmukh8926]

thank you this helped me.

[@davidef.982]

Awesome, thanks 👍

[@jirityr]

Docker Desktop is primarily a virtualization layer which makes it possible to run Docker containers on Windows and Mac. It only exists because you cannot run Docker natively on those platforms. Docker Desktop "doesn't work" on Linux or better to say it doesn't exist for Linux because Docker can run natively there. Minikube is just a way how to run Kubernetes inside a VM (and you can have multi-node cluster and you can run multiple clusters on one machine via Minikube Profiles). Anyway, KinD (stands for "Kubernetes in Docker") is good, but K3D is better. Not only that it's using smaller images and uses less memory (it's using K3S underneath) but it also starts faster, provides more features (e.g. you can stop & start a cluster) and simplifies cluster configuration (e.g. setup Ingress controller requires no crazy configuration).

[@DevOpsToolkit]

You're right about minikube. It can be multi-node multi-cluster solution. Those, however, were never particularly useful for me. I even forgot that they exist. Still, forgetting is not an excuse and I should have mentioned it in the video. k3d video is coming :)

[@lightspeed79]

I run windows, but also have an VM running Ubuntu, thinking of running Kind on this Ubuntu VM.

[@alessandro-affinito]

I'm just about to create a local cluster to clone a production cluster with a different version. It's important to note minikube is not useful for real testing due to api limitations. I was going to use a custom kind install, but now I will search for k3s :)

[@shruh493]

Thank you so much

[@DevOpsToolkit]

You're welcome

[@BandarAlmarri-l2i]

minikube for macOS chips is just hard you need to know ports forwarding , dns resolving etc to just open something on the browser also you need to configure core dns configuration in k8s just too difficult

[@chongerwin]

Thanks!

[@DevOpsToolkit]

Thanks a ton!

[@cloudevops]

What's the watch model on your wrist? :) Thanks!

[@DevOpsToolkit]

It's Apple Watch.

[@santoshkaluskar5565]

Thanks for the video, I really lol your short videos - clear to the point. I am a beginner in Kubernetes space and would like to know what's the difference between k3d and Kind? TIA

[@DevOpsToolkit]

k3s is a Kubernetes distribution designed for edge. As a result, it is more lightweight, requires less resources, is faster, etc. k3d is a distribution of k3s designed to run inside Docker and, in my opinion, is better than kind. That being said, today I prefer Rancher Desktop which is also using k3s, but is a full replacement for Docker. It has everything you need.

[@santoshkaluskar5565]

@@DevOpsToolkitThanks for the reply. I was getting started with Fluxcd (referring your video) and it recommends Kind. And, I was using k3d. I was getting some errors while downloading manifests. And was recommended for Kind, as this had to with some issue with CNI.

[@DevOpsToolkit]

@@santoshkaluskar5565 Software industry changes too fast. Many of my recommendations from a year ago are not valid any more. I strongly recommend Rancher Desktop today.

[@santoshkaluskar5565]

@@DevOpsToolkit would that be compatible with Fluxcd and Linkerd. As these are the projects Iam working with. Plz rec a compatible tool.

[@DevOpsToolkit]

First of all, sorry for not responding earlier. KZbin moved your last message to spam (no idea why) and I did not see it until now. Rancher Desktop is a valid Kubernetes distribution so anything working in Kubernetes in general (including Flux and LinkerD) works in it as well.

[@Roman-zs9qm]

Hi Viktor! Your videos are awesome as well as your courses on Udemy. You've inspired me to try and set up a local Kubernetes cluster on my laptop for local testing and development. I still haven't gone through all the possible options out there and to be honest it's a bit overwhelming. My current setup is Docker Desktop with WSL2 as a backend (Ubuntu 20.04), could you recommend what would be the go to choice? Skaffold? Kind? K3d?

[@DevOpsToolkit]

If you already have docker desktop, that is more than good to begin with. There's no need for alternative local Kubernetes clusters.

[@ioannispolyzos1160]

Cool Video, just came across the channel and really loved it.Btw what software do u use for screen recording?

[@DevOpsToolkit]

I'm using OBS for all the recordings.

[@Mutombo71]

there is now podman desktop

[@portiseremacunix]

Thanks. If I have several linux VM and I want to create K8s clusters using transform these linux VM. Should I use kind or others?

[@DevOpsToolkit]

Use KinD only for ephemeral clusters (e.g., dev or testing environment). Do NOT use it for anything serious like permanent environments (e.g., production, staging, etc.). KinD is a Kubernetes cluster running inside a container. Kubernetes was not designed to run like that. Also, I don't think it can run across multiple nodes (VMs).

[@portiseremacunix]

@@DevOpsToolkit Thanks for your reply! I would like to learn how to set up a dev cluster with networked VMs. It seems that KIND is not the answer. :)

[@DevOpsToolkit]

You can use rancher or tanzu if you prefer a GUI approach or k3s if you don't and you want something light.

[@joebowbeer]

I think the primary dimension ("sorting column") for these comparisons should be the platform. For example I suggest a macos-specific comparison between minikube, kind and k3d. The problems on each platform are different. On macos, ingress is the biggest hurdle. Multi-node, if needed is next. Some tools can do multi-node, but effectively only a max of 2. Trying to explain all this at each level for each separate platform is too confusing to follow.

[@DevOpsToolkit]

That makes sense. Adding it to my TODO list... :)

[@georgelza]

@@DevOpsToolkit would also love to see a K3d vs Kind play off (macOS platform)

[@DevOpsToolkit]

@@georgelza Both would loose. I believe that Rancher Desktop is the best option today. It's also based on k3s but, unlike k3d, it works as a standalone (no need to Docker VM). kzbin.info/www/bejne/mIq5l3ambsmFiqc kzbin.info/www/bejne/m6e6gZyYZc6Bnbs

[@georgelza]

@@DevOpsToolkit I actually like the fact that it's done inside K3d because of the inside docker,

[@DevOpsToolkit]

@@georgelza Within those that work inside Docker, k3d is, without doubt, the best option.

[@niharpatel2631]

Why not kind on production? It is iaac, allows ability of multi node, faster. What are the real cluster you mentioned? How kind compares?

[@DevOpsToolkit]

Kind runs a cluster inside a container. As a result, you'll be running containers inside containers and that is likely going to cause performance issues, networking issues, etc. Those are not visible when running locally. The question is what you're trying to accomplish if not using "real" clusters like GKE, EKS, RKE, etc. Are you trying to have many small clusters so having many control planes is too much of an overhead? If that's the case, try vCluster. Do you want a more lightweight distribution of Kubernetes? Try k3s. Is it on-prem and you want immutable way of managing servers? Try Talos. If you can let me know the objectives/issues, I might be able to get back to you with a suggestion. For now, all I can say is "do not run containers inside containers in production". As a side note, I do not use kind any more. I prefer rancher Desktop for local Kubernetes. P.S. You'll find videos on this channel for all the tools I mentioned in this comment

[@niharpatel2631]

Sure Viktor, I am Looking to run production grade multi node kubernetes in bare metal linux server and single node in client pcs (windows/Linux), looking to package client side cluster for faster client onboarding. Any thoughts?

[@DevOpsToolkit]

Try Talos, k3s, and RKE.

[@adilbsibiss7303]

Thanks for video, what do you think about jenkins x on on promise cloud ?

[@DevOpsToolkit]

I worked as a PM on Jenkins X but, since I moved to other challenges, I stopped following it. So, I am not up-to-date with it and cannot say what shape it is in.

[@JOJO-oe7og]

Hello brother, tried to install tekton on k3d had some issues with it, but with kind it managed to install properly. Why is this?

[@DevOpsToolkit]

Not sure... It's been a while since the last time I used k3d (I switched to Rancher Desktop exclusively). My best guess, without taking a look at the logs and events, is that Tekton does not like running inside a cluster that runs as a container.

[@JOJO-oe7og]

Thank you for the quick reply, love your videos really helped me out with k8s in my DevOps journey. I will look into this 👍

[@sebastijanp3]

if i need to run kind on wsl2 ubuntu and i must not have docker desktop installed, will it work? as i understand all i need is simple docker installed on wsl ubuntu?

[@sebastijanp3]

i see now, you were talking about it at the end. So if i am not allowed to use docker desktop on wins, then i am in position of only having linux on wsl, and that means i should use kind with only docker installed on linux and it should work.

[@DevOpsToolkit]

Yeah. KinD needs Docker. In Linux that's docker alone and in macOS and windows it's docker desktop that creates a Linux VM since Docker alone cannot run natively.

[@tsigian1]

Great! You think Kind can be used for integration/unit testing with GitOps ? (new video?)

[@DevOpsToolkit]

It can. It is especially useful on-prem where spinning short-term cluster might be challenging.

[@michalk2497]

Hi personally I prefer k3d, because kind has a lot more memory requirements. For E2E testing on k3d check github.com/AbsaOSS/k3d-action

[@DevOpsToolkit]

Nice. I'll try it out.

[@rockstarjazzcat]

Helpful video! I’m having difficulty writing from Kubernetes linux clusters on Docker Desktop Windows dev machines, to a mapped network drive. I need the Volumes and Persistent Volumes to work locally. Deep and dark rabbit hole! I’m wondering if a switch to Kind will help? Thanks in advance.

[@DevOpsToolkit]

I don't think that mounting volumes with code is a good idea. You should treat your cluster as remote, no matter whether it is running in the same machine as the one you use for a development or somewhere else. Instead of mounting volumes with code (or whatever you need for development), I prefer synchronizing data. I would suggest checking out Skaffold (kzbin.info/www/bejne/p4TCZYScbdSqmZI), DevSpace (kzbin.info/www/bejne/pILPqpJ5es99mZY), and similar tools.

[@rockstarjazzcat]

@@DevOpsToolkit Thanks! I will take a look at those options, though I'm not sure that we are talking about the same thing. I need to write output from my pods to a Windows server that won't be moved or reconfigured. I'll have the same issue when I go to production, so I'm working out the best way to handle this.

[@DevOpsToolkit]

@@rockstarjazzcat We indeed probably did not understand each other. I thought that you want to mount a volume with your source code during development. Now I see that it's something else, but I'm not sure what it is. Can you explain a bit more what you're trying to do? I'll do my best to come up with a suggestion...

[@DevOpsToolkit]

Which one are you using to run Kubernetes locally?

[@igorfrezer6642]

I used minikube on Windows and Linux for 2 year for some local test, but kind - that good tool

[@benoitbelair3749]

Thank you Viktor for doing this video ! You don't have to convince me to use kind because i am using it on my local laptop inside of WSL2 on Windows 10 since 8 month now ! It's worked fine and i love it ! I don't use anymore Minikube, Vagrant stuff, etc. to install Kubernetes locally ! I switch to WSL2 when i bought your book The DevOps Toolkit Catalog, Patterns, And Blueprints. Continue your good work !

[@DevOpsToolkit]

Great to hear that!

[@caesarion6]

kind is great. the config file is awesome since I can add the features I want, example admission webhook. A feature request for video: Nomad from Hashicorp.

[@DevOpsToolkit]

Nomad added to the TODO list...

[@MegaMario0007]

On a Linux host would you prefer Kind or K3d? (or anything else)

[@DevOpsToolkit]

I'd go with k3d.

[@georgelza]

would you mind sharing you nginx ingress configuration for ingress. I'm bouncing between Kind and nginx ingress or switching to K3d (on my Intel MAC... keen to hear if there is any issues coming along with the new ARM platform, have all of this been ported by Docker as Kind and K3D run inside Docker). my long term target for all scripts/deploys are AWS so need to write/simulate with that in mind.

[@DevOpsToolkit]

If you switch to k3d, it already comes with traefik ingress which you can access through 127.0.0.1

[@georgelza]

@@DevOpsToolkit now to figure out how to configure this... ;)

[@DevOpsToolkit]

@@georgelza Something like k3d.io/v5.0.1/usage/exposing_services/?

[@georgelza]

@@DevOpsToolkit been looking at that exact doc. Ran into a error during that...

[@DevOpsToolkit]

@@georgelza I do not have Docker running any more, but I can set it up (probably over the weekend) and get to you with the steps if you do not figure it out by then.

[@ahmedmk534]

at which point you deployed your application (web app) ?

[@DevOpsToolkit]

It's been a while since I created that video so I don't remember the details. Most likely i did not deploy an app since that is not directly related to kind. An app would run (more or less) the same no matter which flavor of kubernetes is used so it's not necessarily relevant for that video.

[@EdwardPike]

Kind on WSL2?

[@DevOpsToolkit]

Probably. It should work in Docker running anywhere. That being said, I do not own a Windows machine anymore so I cannot check it.

[@anshuman2121]

Hey I’m looking to setup local dns xip.io what you usually do in most of the videos.

[@DevOpsToolkit]

xip.io stopped working a few weeks ago so I switched to nip.io instead (and changed all the gists). nip.io (as well as xip.io while it was working) are not local DNS. They are public services that forward requests to the IP specified as a subdomain. So, if you use 1.2.3.4.nip.io, requests to nip.io will be forwarded to the IP 1.2.3.4. That way you get a free domain (nip.io) without having to configure DNSes. It's very useful for local development, testing, and other use cases when setting up a real domain might be too much of a hustle.

[@anshuman2121]

Fully understand this now. Thanks. 1 last question I have. I am thinking to switch from Jenkins. It there any other cloud native cicd tool out there in which I can integrate SonarQube for code test before build and deploy.

[@DevOpsToolkit]

Integrations are not an issue anymore anywhere. All tools today have an API so you can send a request to it to do whatever you need to do. Similarly, there is a container image for everything. From that perspective, any pipeline tool should do. You might take a look at videos about tekton, Argo workflows, and GitHub Actions on this channel and check whether any of those suits your needs.

[@erickbrenes7562]

First like :p

[@joebowbeer]

minikube --nodes 2 creates multi-node cluster

[@joebowbeer]

minikube.sigs.k8s.io/docs/tutorials/multi_node/

[@DevOpsToolkit]

That was a mistake on my part (which I think I corrected in one of the comments). I've been using minikube for such a long time that I missed that, since then, it introduced the --nodes argument. My bad.

[@AndreasBrenk]

What about minikube's Docker driver (minikube.sigs.k8s.io/docs/drivers/)? That's minikube on Linux without a VM. And there's also the experimental podman driver.

[@DevOpsToolkit]

From my experience, if you do want to run k8s as containers, minikube is too clunky and too heavy. Also, it suffers from trying too many modes (it can be a VM, it can be a container, etc.) which might be preventing it from being exceptionally good in any. kind is better if what you need is to run k8s as containers. k3d might be even better option. I'm working on a video about it, so I'm trying not to give away any spoilers.

[@DevOpsToolkit]

... but you are right that I did not mention that, and I should have.

[@oraculotube]

how can I do a like twice :)

[@alessandro-affinito]

In these days I had to replicate an old K8S cluster (1.15) on a local Fedora. Since I didn't find the image for K3S I used Kind. I had to configure some persistent volume (manual hostPath), multi node, calico CNI. It was a lot harder than expected and still I have to solve some issues.. The config file has very few options to use and I had to put the image version as argument (removing it from the config where it was ignored) and couldn't use a --kubeconfig arg, because it leads to certificate mismatch error. Also pod CIDR seems to be ignored from config file (172.28.0.0/16 instead of 192.168.0.0/20). gist.github.com/tuxerrante/ecd35edcf549bc03ba6493e05c9ebe8f Maybe for complex local tests does exist some better solution, less complex than kubespray?

[@DevOpsToolkit]

Things can easily get messy with kind, especially if you have "special" requirements for the OS (e.g., Fedora).

[@alessandro-affinito]

@@DevOpsToolkit which is the most compatibile distro? Ubuntu?

[@DevOpsToolkit]

@@alessandro-affinito I don't think that any k8s running in containers will give you that. If you need to have the same (or similar) OS then your best bet might be Vagrant. On the other hand, if OS is not that important, you could create kind images using whichever OS you need. Just remember that OS in containers is not the same (or even close) to OS running on nodes, especially if those nodes are supposed to run contains with Docker (which cannot run in containers).

[@sixteenornumber]

`kubectl get nodes` --> The connection to the server localhost:8080 was refused. I also tried running `kubectl get nodes --server localhost:6443` ---> "Unable to connect to the server: EOF" ::::EDIT::: I found the problem, I had to re-export the config. export KUBECONFIG=~/.kube/config

[@DevOpsToolkit]

Is it a company laptop? Can there be some VPN/firewall/etc. that might be blocking the communication?