How to Debug Kubernetes Applications With Ephemeral Containers

[@DevOpsToolkit]

How do you debug containers in Kubernetes clusters when observability tools are not enough?

[@hugolopes5604]

breakglass, and ssh into the container, and do a post mortem for the breakglass

[@chrisre2751]

Thanks!

[@DevOpsToolkit]

Thanks a ton!

[@maxreuv]

Another one of "must haves". Thank you!

[@hkupty]

Thanks for that! I have a label in the pods that is `live = 1`, which I use for taking pods out if want to debug them (setting `live = 0` would cause deployment to spawn a new pod and service to route to the newly created pod). I can safely run kubectl debug on that pod now given it can safely be removed later anyways :)

[@javisartdesign]

thanks! really useful, sometimes it's necessary to test communications and other stuff

[@IvanRizzante]

Thanks for another great video! I totally agree on using ephemeral containers for debugging, but for GitOps lovers like me that kubectl debug means that I have to contact Joe, that guy that works in the basement and nobody knows...who's the only one having direct access to the cluster 😅

[@DevOpsToolkit]

Not really... Kubectl debug command is just a shortcut that contacts kube api to edit the pod definition. You could modify your manifest by adding ephemeral container spec, push it to git, and get the same result. From there on it all depends on whether you have permissions to sh into that container.

[@IvanRizzante]

@@DevOpsToolkit that's what I thought too, but as you said that would require kubectl exec in the end. I agree with you when you say that even in GitOps world you still need someone authorized to interact directly with the cluster, so I think that Joe would save my day in these corner cases!

[@DevOpsToolkit]

To me, ephemeral containers are the last resort when all observability tools )that do not need kubectl) fail to reveal the issue. That's the "break glass" situation which always requires elevated permissions. It's "we do not know what to do so let's dig in" type of situation.

[@IvanRizzante]

@@DevOpsToolkit I totally agree 👍 now I'm waiting for a video on kubearmor 👍

[@devxp-tech]

Very helpful, thank you! ❤

[@toddfinoch]

Curious to know if you ever had a scenario where the debug pod worked but the original did not still? Just an example, you have a DNS resolution issue and want to do an nslookup, but don't have it installed in the original container.

[@DevOpsToolkit]

Most of the time i don't have anything but the app itself in original containers so the only way to debug issues that cannot be observed from outside pods is to add a debug pod.

[@toddfinoch]

​@@DevOpsToolkit thanks for the quick reply. Could it be possible that the problem is only in the container?

[@DevOpsToolkit]

@toddfinoch i don't have any info so anything could be a problem.

[@Babbili]

i keep on saving your videos 😆

[@chrisre2751]

Hi Victor, how do you think about distroless-images from google?

[@DevOpsToolkit]

I used them only briefly so I'm not th Best judge of them. What i saw makes me think they're ok.

[@devopssimplify9884]

Again a great Video!!! I wanted to implement the same but the problem is my cluster has restriction for specifying resource limit and request for pod. How to mention that with kubectl debug? My debug pod needs to have resource mentioned, please suggest!!!!

[@DevOpsToolkit]

The `kubectl debug` command does not allow you to specify resource limits and requests. As a matter of fact, `kubectl create` does not have that option either. The logic behind `kubectl` is that it provides only some of the flags when creating resources. The assumption is that anything more needs to be specified as a YAML file. That is equally valid for `kubectl run` as for `kubectl debug`. What that means is that you'd need to define emphemeral containers as part of your manifest if you need resource limits and/or requests. The spec is in: kubernetes.io/docs/reference/generated/kubernetes-api/v1.27/#ephemeralcontainer-v1-core So, when you want to debug, add ephemeral container to manifest with the Pod (or Deployment or StatefulSet) and execute `kubectl apply --filename ...` or whatever you're using normally.

[@DevOpsToolkit]

The alternative solution is to use Mutating Admission Controllers with, let's say, Kyverno. You can configure it to add resource requests and limit every time an ephemeral container is created. That is probably a more elegant and user-friendly solution.

[@devopssimplify9884]

@@DevOpsToolkit Wow wonderful! thanks a lot!

[@devopssimplify9884]

@@DevOpsToolkit You are really genious!

[@lightman8527]

Great video! On a side note, never heard about kubearmor. Are you planning to make a video on it? How mature is the project in your experience?

[@DevOpsToolkit]

KubeArmor is on my to-do list. That's why I mentioned it.

[@Babbili]

@@DevOpsToolkit is it an alternative to Falco Sysdig ?

[@DevOpsToolkit]

It is :)

[@squalazzo]

so, it seems like this is not a solution for scenarios like a statefulset with mongodb, with 2 replicas and 1 headless pod for backup, for example... or any statefulset, in the end, as you're working with data which could be already old, and not lively used by the app using that sts... right?

[@DevOpsToolkit]

Debugging by entering containers is (should be) the last resort used only when observability fails to show what we're looking for. On top of that, we rarely debug third-party apps like that. That's typically reserved for our own apps. Now, if you do need to debug (like that) third-party apps, and they are stateful, and their containers do not have the tools we need (which they often do), I tend to NOT create a copy but attach ephemeral container directly to one of the STS pods.

[@squalazzo]

@@DevOpsToolkit thanks!

[@abcd-z9u]

I tried this, but my debug pod is failing with the following error i got from describe: message: '0/7 nodes are available: 1 node(s) didn''t have free ports for the requested pod ports. preemption: 0/7 nodes are available: 1 node(s) didn''t have free ports for the requested pod ports, 6 node(s) didn''t match Pod''s node affinity/selector.'

[@GoshaDo]

debugging in production is safe. We are doing it all the time using Rookout dynamic observability tool. much more advanaced tech for debugging then attach containers and stoping the application.

[@chasim1982]

Great 👍 Content & Learning, one question can I do same stuff by launching a new pod of my application with debug side car? Please correct me, if I am wrong, thanks for amazing content, learned a lot from you ❤.

[@DevOpsToolkit]

You can create a new pod and then attach a debug container but i think it's easier to simply use the --copy arhument.

[@chasim1982]

@@DevOpsToolkit I am using eks 1.23

[@bhavasarpritesh]

How can we see pods files in debug container?

[@DevOpsToolkit]

la commabd should work

[@DennisHaney]

Tbh. If I had to resort to this for debugging, the first thing i would afterwards was to delete the pod to see if it came back with no errors

[@DevOpsToolkit]

Yeah. That's why creating a copy of a pod is very useful.

[@oleksandrsimonov9200]

Still the question how to make a such limited container with Node, Python or Ruby? Ok, you don’t have a shell, but you need an interpreter

[@DevOpsToolkit]

For those, i would pick ChainGuard images. They are small, (mostly) without vulnerabilities, abd built daily. You'll find a variation for almost every language.

[@oleksandrsimonov9200]

@@DevOpsToolkit yeah... and they using alpine... Its causes a lot of issues with ffi for example, where we need to use glibc instead. For now we using a mulsistage minideb images with removing of dev packages and other not needed things on latest stage.

[@DevOpsToolkit]

I thought that ChainGuard fixed that by using glibc instead of musl.

[@jemag]

​@@oleksandrsimonov9200they use Wolfi, not alpine. You can use glibc or musl

[@PS-xn8ej]

you first should show kubectl get pods on what you are working, not end command ...

[@testtestlast4306]

👀

[@iximiuz]

While I fully aggree with the overal sentiment, I'm afraid this video is dangerously misleading in details. 1) Don't build your container images `FROM scratch` - scratch containers lack the vital parts of an OS like /etc/{passwd,group} files and TLS certificates (if you absolutely have to use a minimal image, use a slim/distroless base) 2) Ephemeral containers don't run forever - as soon as the main process exits (of crashes), the ephemeral container stops.

[@petermerelis]

this is why Dockerfiles should be multi-stage. build with all necessary bits and then copy to as slimmed down an image as possible for the runtime.