Deploying machine learning models on Kubernetes

[@ludwigstumpp]

Always a pleasure to watch someone as talented as you! Keep it up :)

[@mildlyoverfitted]

Wow, much appreciated:) Thanks:)

[@abdjanshvamdjsj]

Brooooo this was so good.

[@mildlyoverfitted]

Glad you liked it!

[@vishalgoklani]

Welcome back, we missed you!

[@mildlyoverfitted]

Hehe, thank you! Nice to hear that:)

[@alivecoding4995]

I agree!

[@humanity-indian]

Great example. Thanks for the information

[@mildlyoverfitted]

My pleasure!

[@fizipcfx]

he is back 🎉

[@kwang-jebaeg2460]

OH !!!!! Glad to meet you again !!!!

[@mildlyoverfitted]

Glad you are here:))

[@shivendrasingh9759]

Really helpful for foundation on ml ops

[@mildlyoverfitted]

Glad to hear that!

[@thinkman2137]

Thank you for detail tutorial!

[@thinkman2137]

But torchserve now has kubernetes intergration

[@mildlyoverfitted]

I will definitely look into it:) Thank you for pointing it out!!

[@ivanxiecornell]

Would appreciate a video using VScode to include docker contain files, k8s file and Fast API

[@JoseMiguel_____]

You're great. Thanks for sharing this in such a nice way.

[@mildlyoverfitted]

My pleasure!

[@aditya_01]

great video thanks a lot really liked the explanation !!!.

[@mildlyoverfitted]

Glad it was helpful!

[@unaibox1350]

Amazing video. In min 5:25 how did you do to open the second bash in the console? I was searching for a long time and I can't find anything. Thanks and regards!

[@mildlyoverfitted]

Thank you! You need to install a tool called tmux. One of its features is that you can have multiple panes on a single screen.

[@unaibox1350]

@@mildlyoverfitted Thank you! Will dig in it now

[@davidyates4857]

Great video very informative.

[@mildlyoverfitted]

Glad you liked it!

[@SmiteLax]

Cheers mate!

[@maksim3285]

Thank you, it helped me a lot .

[@mildlyoverfitted]

Happy to hear that!

[@bpac90]

excellent!! I'm curious why my search always shows garbage and videos like this never come up. This was suggested by Gemini when I asked a question about ML model deployment.

[@zhijunchen1248]

Hi, I would like to use GPU to accelerate this demo, can you give me some tips? Thank you

[@mildlyoverfitted]

So if you wanna use minikube this seems to be the solution. minikube.sigs.k8s.io/docs/handbook/addons/nvidia/

[@zhijunchen1248]

@@mildlyoverfitted thankyou, i use the "--device" flag of transformers-cli to enable GPU. And I found that serving app takes up almost gpu memory and no compute power. Whatever, thankyou for your video!

[@unaibox1350]

I am having a problem in the min 18:00 the model load is being killed all the time. I tried to "minikube config set memory 4096" but still having the same problem. Any idea? I've been looking for a solution for 3 hours and there is no way

[@mildlyoverfitted]

Hm, I haven't had that problem myself. However, yeh, it might be related to the lack of memory.

[@davidpratr]

really nice video. Would you see any benefit of using the deployment in a single node with M1 chip? I'd say somehow yes because an inference might not be taking all the CPU of the M1 chip, but how about scaling the model in terms of RAM? one of those models might take 4-7GB of RAM which makes up to 21GB of RAM only for 3 pods. What's you opinion on that?

[@mildlyoverfitted]

Glad you liked the video! Honestly, I filmed the video on my M1 using minikube mostly because of convenience. But on real projects I have always worked with K8s clusters that had multiple nodes. So I cannot really advocate for the single node setup other than for learning purposes.

[@davidpratr]

@@mildlyoverfittedgot it. So, very likely more petitions could be resolved at the same time but with a very limited scalability and probably with performance loss. By the way, what are those fancy combos with the terminal? is it tmux?

[@mildlyoverfitted]

@@davidpratr interesting:) yes, it is tmux:)

[@alivecoding4995]

What terminal application is this, with the different panels?

[@mildlyoverfitted]

tmux

[@lauraennature]

New video 🤩

[@johanngerberding5956]

very cool video!

[@mildlyoverfitted]

Thank you! Cheers!

[@evab.7980]

👏👏👏

[@EvaKreplova]

Great!

[@nehetnehet8109]

Great

[@nehetnehet8109]

Realy goood

[@kwang-jebaeg2460]

Look forward to show your face alot :))

[@SunilSamson-w2l]

the reason you got . , ? as the output for [MASK] because you didn't end your input request with a full stop. Bert Masking Models should be passed that way. "my name is [MASK]." should have been your request.