GPUs in Kubernetes for AI Workloads

Today we dive into running AI models on Kubernetes with GPU support. Learn how to manage GPUs in Kubernetes clusters, create GPU nodes, and optimize resource usage without breaking the bank. We'll walk you through setting up a Google Cloud Kubernetes cluster (the same logic should apply to other Cloud providers), deploying AI models like Ollama's Llama2, and handling GPU partitioning. Watch now to master GPU-based AI workloads in Kubernetes!
#Kubernetes #GPU #AI
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Sponsor: CAST AI
🔗 cast.ai
▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬▬
Consider joining the channel: kzbin.infojoin
▬▬▬▬▬▬ 🔗 Additional Info 🔗 ▬▬▬▬▬▬
➡ Transcript and commands: devopstoolkit.live/ai/unlock-the-power-of-gpus-in-kubernetes-for-ai-workloads
▬▬▬▬▬▬ 💰 Sponsorships 💰 ▬▬▬▬▬▬
If you are interested in sponsoring this channel, please visit devopstoolkit.live/sponsor for more information. Alternatively, feel free to contact me over Twitter or LinkedIn (see below).
▬▬▬▬▬▬ 👋 Contact me 👋 ▬▬▬▬▬▬
➡ Twitter: vfarcic
➡ LinkedIn: www.linkedin.com/in/viktorfarcic/
▬▬▬▬▬▬ 🚀 Other Channels 🚀 ▬▬▬▬▬▬
🎤 Podcast: www.devopsparadox.com/
💬 Live streams: kzbin.info
▬▬▬▬▬▬ ⏱ Timecodes ⏱ ▬▬▬▬▬▬
00:00 AI Inference with GPUs
01:30 CAST AI (sponsor)
02:29 Using GPUs for AI Inference in Kubernetes

[@thibdub2752]

Hello Viktor, thank for the video, I would like to see a video with Knative and KubeVirt.

[@maaft]

Hi Viktor, glad that you mentioned KubeVirt. I did not find a solution yet where one can have on-premise (or cloud-provider A) GPU Servers to cover the base-load, while at the same time scaling into another cloud-provider B when there is demand. I coldnt get it to work yet and tried different solutions (Admiral, KubeVirt, Karmada) but there was always one or more roadblockers. Most of the time, the scheduler would just not even try to schedule my workload since all GPUs are already used. But IF the scheduler would just go ahead and schedule, auto-scaling would have picked up and spawn a new GPU node. This topic could also be expanded on the general case of how to do multi-cluster workload distribution (with auto-scaling) As always, thanks so much for your valuable content!

[@mgroth3973]

Great content as always! As someone who works a lot with KServe, I would of course like to see a video clip about your preferred approach of scaling InferenceServices in prod.

[@dariusjuodokas9458]

yes, yes, please make a video about models. Also it would be nice to have a concise "Intro" into our options (tools) to train and/or fine-tune our own self-hosted models (or managed?). I want to dip my fingers into ML, but the technology is so fast growing and changing, that it's quite difficult to get 3 consistent search results (videos/blogs) explaining what is what and how to DIY. And ML seems to be the way to go further, as it seems to be rather good at automation w/o the need to write custom software., so we DO have to master it

[@Sebastian-or4xw]

Just FYI: a single Ollama installation and instance can run multiple models. Of course after each other when one is no longer used, but if you have enough VRAM also at the same time now

[@DevOpsToolkit]

That's true. I installed it twice only to demonstrate how sharing gpu works.

[@mrgdevops]

not just ollama... differentiate bwn few llms .... helpful in devops space

[@zuduni]

Kubevirt with gpu will be cool, thx

[@nemethk]

Pozdrav! A video of Ollama AI models in your presentation would be a valuable content.

[@DevOpsToolkit]

Great. I'll add it to my to-do list.

[@ovsepavakian6109]

Also you can use Slurm Workload Manager or Volcano

[@theindependentradio]

Yes please show the knative option

[@Fayaz-Rehman]

Thank you sharing - Could it be possible if I can apply the same locally on my homelab ?

[@DevOpsToolkit]

If you have gpu in your homeland, yes you can. Th setup will be more complicated though.

[@TiggyProph]

What are your recommended tools to manage GPU workloads on Kubernetes? At my org, we've configured the basics that you have here already, and now the teams are looking into AI frameworks. (Using Argo, Karpenter, and EKS to manage all the configurations discussed in your video here) Applications like Kubeflow are being discussed to help those teams move more swiftly, and I'm curious about your take on it or if you have content coming soon related to that.

[@DevOpsToolkit]

I explored inference. Kubeflow is focused more on generating models. It is great, but sometimes overwhelming. I'll do my best to explore it in one of the upcoming videos.

[@Anselm_0]

Yes, an orchestrator definitely makes your life easier and there are pretty good open source ones. Airflow, prefect or flyte are pretty good. From Kubeflow I've heard mixed experiences.

[@chandup]

zenML could help make life easy to use many ML tools.

[@Sebastian-or4xw]

Kubeflow is the ML everything tool (collection), but it is not very easy to deploy and maintain in my experience. We used deployKf which made it a bit easier but that does not include everything

[@renanmonteirobarbosa8129]

What if my model requires 32 gpus to perform inference? 😜 what see your K8s do that

[@DevOpsToolkit]

What would you use instead?

[@renanmonteirobarbosa8129]

@@DevOpsToolkit SLURM as reccomended by Nvidia. They have lots of eduational material on the top and recently updated DLI labs