How To Troubleshoot And Support Kubernetes Applications And Clusters?

[@DevOpsToolkit]

Do you provide support or help out others when they face problems with their apps or clusters? If you do, what do you use?

[@shuc1935]

Hi Viktor, another gem, really enjoyed your pedagogical mentoring like always. I have a stupid question - is the installation of preflight using krew as k8s plugin NOT conflicting the claim that tool is not intrusive? May be you have different perspective of intrusive instrumentation but on the face value we are mandating installation of plugin in the production cluster which may not be approved by the enterprise security policy although the information is not leaking out of enterprise. Just curious stupid question , please pardon me in case I have misconstrued the idea.

[@DevOpsToolkit]

@@shuc1935 I probably did not explain well what I meant by "non intrusive". I wanted to say that nothing is installed in clusters (especially not in production). Nevertheless, you do need to install the CLI plugin. Or, even better, you might want to have it running in your pipelines or something similar where you can control what is collected. In any case, you do need to change the state of your laptop (or some dev server or pipelines or ...), but not of any of the "real" servers/clusters. > installation of plugin in the production cluster which may not be approved by the enterprise security policy although the information is not leaking out of enterprise. It's not really installation of anything in production, but on your laptop or wherever you're having `kubectl`.

[@shuc1935]

Very nicely addressed. My bad , it was not the right question, I agree completely with you now realizing it was a genuine stupid question, my sincere apologies , I will be careful next time not to repeat.

[@DevOpsToolkit]

Please do not say "stupid question". If anything, it can only mean that I did not explain it well, not that you asked a stupid question.

[@shuc1935]

@@DevOpsToolkit This is why you are situated a different plane. Learning from you.

[@JaydeepDave12]

Best channel in the field of DevOps.

[@DevOpsToolkit]

Thanks Dave

[@galonge]

Amazing, thanks Victor!

[@fenarRH]

If you are running enterprise apps on enterprise grade k8s distro backed with proper sla, most likely "oc adm must-gather" may suffice for collecting necessary data for trouble-shooting.

[@bombaclotta]

oc is for a very specific K8s platform. Hmmm.

[@fenarRH]

@@bombaclotta Lars that is true

[@spottedmahn]

Where would they get the preflight or support bundle yaml files from? 🤔

[@DevOpsToolkit]

That depends who you're referring to. People providing help/support would provide the definitions and people in need of help would run them through the CLI and send the results

[@spottedmahn]

@@DevOpsToolkit I see, thanks! So there’s no fancy way of sharing those files & ensuring they have the latest, right? 🤔

[@DevOpsToolkit]

@@spottedmahn I tend to just keep all the files (including those you mentioned) in Git repos. Among other things, that ensures that people have the access to the latest versions (e.g., mainline).

[@javisartdesign]

Thanks, very cool tool. I would like to know other (more automatic) alternatives such as kube Hunter, Kube Bench, MKIT etc.. Troubleshoot is more Gatekeeper/OPA oriented since you need to configure each policies or rules to be checked.

[@DevOpsToolkit]

I hope that I'll soon be able to make a video about those as well as a comparison. Until then, only a TLDR; The goal of Troubleshoot is a bit different from those. It's supposed to: 1. Help an end-user troubleshoot an issue related to an application provided by someone else. 2. If 1. failed to fix the issue, provide a support bundle to the team or a company responsible for that app (e.g., a software vendor or a different team in the company). In both cases, the goal is to be able to trouble-shoot and collect data in clusters that we do not own or, likely, cannot even access directly. It's about support/end-user relationship. Now, for something like that to work, it must be very configurable since it's not about generic stuff but whether all the requirements for a specific app are met and, if they are, about collecting info about a specific app. Does that make sense?

[@javisartdesign]

@@DevOpsToolkit Sure. What I was able to try testing the other tools, those provide a "kind of" static analysis to your kubernetes cluster based on best practices and known vulnerabilities, similar to Quality Source Code tools like Sonarqube or kiuwan.

[@DevOpsToolkit]

@@javisartdesign Yeah. That's kind of the main difference. Those you listed are great for people who want to validate the state of their own clusters, while Troubleshoot is for those who want to provide support (one way or another) without really being in control of the destination. The company that opensourced it is in that kind of a busines. Anyways... I'll do my best to go through "static analysis" type of tools as soon as I can.

[@chandrasekharreddy8177]

Hi this video is very usefull for me. We are using eks cluster for my application, we are upgrading cluster components like clusterautoscaler, Prometheus stack, fluent-bit,..while upgrading the helmchart versions I am errors """crashloopback and imagepullbackoff ''""issues. Could you please help me out this how to resolve?

[@DevOpsToolkit]

Most likely, k8s cannot pull the image, either because it does not exist or because it does not have permissions yo do so. You should describe the pod that is failing to see the events. That should give more info.