VM vs Kubernetes: Performance 🚀

[@AntonPutra]

🔴 To support my channel, I'd like to offer Mentorship/On-the-Job Support/Consulting ([e-mail]) 🍿 Benchmarks: kzbin.info/aero/PLiMWaCMwGJXmcDLvMQeORJ-j_jayKaLVn&si=p-UOaVM_6_SFx52H 👋 AWS is expensive - Infra Support Fund: buymeacoffee.com/antonputra

[@martinhotmann7868]

Aaaah K8s vs Docker vs LXC vs VM would be awesome! :) All on the same HW of course.

[@9inl]

2:22 Would you please do another one with VM vs Kubernetes but with Cilium eBPF networking instead of kube proxy?

[@ayr-ton]

Now we have opened a pandora box. Which is the fastest networking module for Kubernetes? Don't miss the next episodes.

[@vadimlazovskiy8621]

Thanks for the video! Certain flaws in your testing methodology led to the results you've got. First, using ClusterIP service in K8s vs. VM IP is unfair. An IP-address from a service network is allocated for ClusterIP service (without `clusterIP: None` set). In your case your requests traffic goes through some iptables DNAT rules or eBPF program (depending on the CNI used), hence latency increases. Consider running your benchmark with a headless service (clusterIP: None set). Second, what is the purpose of setting CPU limits on a dedicated instance? Hitting cgroups limit is not the same as hogging the entire CPU time available, like you have in case of VM benchmark. Though achieving the same available CPU time in K8s vs. VM is not feasible due to the presence of kubelet and bunch of DaemonSet pods at the K8s instance, you are still capable of not worsening things. Just set the requests.cpu to maximum allocatable value for the node and remove the limits.cpu. Your container's cgroup will have the maximum available cpu.weight and this will have an effect for the process scheduling.

[@dnfrd3187]

K8s clusters as can be found in providers reside on top of VMs. I guess results would be quite different for K8s deployed on bare metal...

[@BakhtiyarGarashov]

I believe VM will still perform better than bare metal k8s cluster.

[@nikolaibarinov8660]

Is it correct to run tests on the different CPU architectures? Gravitron in EKS and AMD in VM?

[@Malix_Labs]

Hey Anton! It would be extremely interesting to see the difference with Talos (Kubernetes as an Operating System, bare metal, not even SSH or anything else in the OS)

[@ramonpereira4460]

What I've seen is that k8s is great for multiple apps, multiple machines, complex applications. For a simple app, I'd always use a VM or even something like AppEngine. Thanks again, Anton.

[@soyangel1]

To complement this maybe you can check vs bare metal. On a provider like Hetzner you can get nice hardware for a little more than 30€/month and no setup fee (check server auction there). I'm not sure if the investment is worth for you, maybe you can run several tests on this server during that month 🤔

[@Antipolitisch]

I think the flexibility and scalability of kubernetes makes this price compared to a vm more than affordable.

[@Antipolitisch]

Can you please redo the test and instanciate another kubernetes instance? I'm interessted if the kubernetes advantage of scalability can work around the fact, that more input output will bring compared to vm

[@ricardopieper11]

So, cloud kubernetes chops off ~20% of your peak performance... good to know.

[@danielh6453]

My two cents: The performance difference in this example is that K8S uses iptables and ipvs to route traffic at the kernel level, whereas a VM has no need to use any abstraction; it can go directly to the endpoint. So although this example has correctly shown K8S is slower, it has really shown that K8S is slower because it uses kernel-level service discover, and if you don't need service discovery, a direct VM is a better choice. This is your point, I think, and demonstrated well.

[@danielh6453]

With that said, if you need service discovery, Kubernetes uses kernel-level performance to do so

[@90hijacked]

Random request, what about benching NATS/JetStream ? It boasts of lots of features comparable to Redis/Rabbit/Kafka/...

[@banglaLang]

Would you please do another one with VM vs Kubernetes but with Go or Rust

[@svetlinzarev3453]

this should definitely be tried with rust

[@sebastiengauthier58]

You should try again with a CPU limit set as an integer, otherwise your pod will have a different QoS class. You need to have a “guaranteed” QoS class. If you want to go further you can set CPU manager policy to static to effectively pin CPU cores which also improves performance in some scenarios.

[@ninele7]

What would happen if you use more performant app in deployment? Maybe Kubernetes overhead would be even more noticeable?

[@franciscopena7859]

A better comparison would be kubevirt vs normal pod. K8s has a lot of resource "virtualization" built into it

[@kamurashev]

Great stuff!

[@Sergey-b2e]

And what about the price? What will be cost effective?

[@pandatv3994]

Postgres SQL on VM vs. Kubernetes

[@toddfisher8248]

now do this with dedicated instances vs vm's

[@maxcoder95]

Knative vs Lambda vs Cloud Run will be a good comparison but will be very expensive 😂

[@jhonbrugg]

app engine também :)

[@iddiiddrisu5971]

Let's go!! VM should eat kube

[@The_real_jamal]

K8s vs swarm 🙂

[@jhonbrugg]

This bugged my mind, how does the vm have better performance than k8s?

[@Grahamaan27]

It's pretty expected if you ask me. K8s has a ton of overhead. A service on a VM is very minimumal from the guest and application perspective (but more from the host perspective)

[@kh0lis]

It is expected for simple apps. K8s use tons of layers...

[@ikhwanrizqynurzaman7579]

Because of the overhead and layers abstraction in k8s. This overhead will become larger if we use larger replicaset. The main major benefit of using k8s is the convenience of deployment and scalability. Especially if your system has tens of applications running

[@deado7282]

Anton is drawing the wrong conclusions from his results. Kubernetes is just API's. It does not implement most stuff by itself. It's much more about the concrete implementations you pick for those API's. Anton is using a pretty standard EKS & applying most of it's defaults. Those are generally not tuned towards performance but convenience. First: EKS runs on top of VM's. So there is no chance of outperforming Applications running directly on those VM's anyway. It's pretty easy to create a Kubernetes Cluster on Bare Metal that outperforms minimal KVM+QEMU VM's, which is one the most performant VM Setup. Even if you combine both by using KQEMU VM's as Nodes for your cluster you would achieve vastly better results than those shown in the video.