Exploring the fastest open source LLM for inferencing and serving | VLLM

Hello everyone welcome to our latest video. Today we will explore the open source inferencing and serving LLM, VLLM. Our founder Vishnu Subramanian will take you through on what is VLLM and also how to use them.
Connect with us on social media:
LinkedIn: / jarvislabsai
Instagram: / jarvislabs.ai
Medium: / jarvislabs
Connect with Vishnu:
X: / vishnuvig
Linkedin: / vishnusubramanian

[@HermesFibonacci]

Very interesting i listened to the very end, and it gave me some ideas for prepping my Model. Thanks for the explanation and demo. May I ask?... Do you think an Nvidia GTX Orin Devkit 64 GB would be fitting for running LLMs locally for fine tuning, training and later deploying to server once developed (both Locally and Server on Ubuntu)?

[@alecd8534]

Thanks for your video. It is interesting. I am new to LLM and one question to ask. When you run JarvisLabs in your demo, does it mean you are running a server running locally to provide API endpoint? Please advise

[@JarvislabsAI]

In the demo, I was running on a gpu powered instance. The vllm server in this case is running in the Jarvislabs instance. You can use the API endpoint from anywhere.

[@alecd8534]

@@JarvislabsAI thanks so much. I have Navida T500 GPU card on my laptop. But it has only 4 gb. Can it run vLLM? Do we need to install JarvislabsAI on our local machine? Does JarvisLab do? Thanks

[@JarvislabsAI]

Not sure, if will be possible to run vllm on T500 GPU. Jarvislabs, offers a gpu instance in which you can use vllm.

[@bernard2735]

This was a nicely paced and clear tutorial. Thank you. Liked and subscribed.

[@JarvislabsAI]

Thanks for the support :)

[@Akshatgiri]

Super useful. Thanks for breaking it down.

[@dineshgaddi1843]

Thank you for sharing this information.

[@JarvislabsAI]

Glad it was helpful!

[@kaiwalya_patil]

An excellent one! Thank you so much for sharing. Any idea about the possibility of fine tuning my own LLM(like Llama/Mistral), uploading back to HF and the put it into production using VLLM?

[@JarvislabsAI]

Yeah definitely possible. Would make one soon.

[@kaiwalya_patil]

@@JarvislabsAI Thank you, looking forward!

[@YajuvendraSinghRawat]

Its a wonderful videa, clearly and concisely explained.

[@JarvislabsAI]

Glad you liked it

[@fxhp1]

hey i also have an AI channel, i tried mistrals model and it didnt finish its execution and looped over the input forever, i had slightly better luck with the instruct version. did you ever get mistral to work?

[@JarvislabsAI]

We tried with vLLM and remember it working. I will probably check again.

[@Ian-fo9vh]

hank you, it was interesting.