How to Use Open Source LLMs in AutoGen Powered by vLLM

[@joeclacher445]

Unreal video! Looking forward to testing various models instead of GPT!

[@yeyulab]

Thanks, would like to see the performances as well.

[@JDWilsonJr]

This is spot on. Thank you for making the video and explaining so well.

[@yeyulab]

Thanks

[@Beenee_AI]

Great ! You are really good at what you do!

[@yeyulab]

Thank you!

[@jaoltr]

Terrific video! Thank you for sharing your knowledge.

[@yeyulab]

Glad it was helpful!

[@KodandocomFaria]

Is it possible to use powerinfer instead of vllm? If possible which one would be faster ? Perhaps a good video to make by comparing those two inference tools

[@yeyulab]

Looks Powerinfer is a pretty new inference tool. It cannot be supported in Autogen directly right now but If you can run Uvicorn to serve its inference, there maybe a chance. Thanks for the recommendation.

[@truliapro7112]

How to use autogen with aws bedrock models ?

[@current.undone]

thanks for sharing. Can vLLM be installed on Mac? Please help if it can as Max Studio has all the musles needed to do the heavylifting 🙂

[@yeyulab]

vLLM does not support MAC backend at the moment and I guess the reason is that they want to maximize the throughput of generation by V100/H100 GPUs.

[@shubhamgarg5007]

Hey, thanks for the in depth explanation. While its great that we can use Autogen along with open source models using vLLM, is there any chance we could use Gemini API along with autogen?

[@yeyulab]

There is an on-going branch of AutoGen working on Gemini integration. Soon you can use it I think. github.com/microsoft/autogen/tree/gemini

[@shubhamgarg5007]

@@yeyulab Yeah, I checked it but it has no commits since the last 2 weeks and I doubt its one of their top priorities as of right now. I couldn' find any online resources to use Gemini's free api with autogen either.

[@yeyulab]

Free Gemini API is really useful I agree. Let me check with their team.

[@Nick_With_A_Stick]

I don’t believe you have it in your youtube tag’s but you should fill our your youtube tags with things like “ vLLM tutorial” as k looked for one and came up very very short, and this would’ve been much mote useful. Thanks for the vid!

[@yeyulab]

Good suggestion, thanks!