Deploy ANY Open-Source LLM with Ollama on an AWS EC2 + GPU in 10 Min (Llama-3.1, Gemma-2 etc.)

[@DevelopersDigest]

The best way to support this channel? Comment, like, and subscribe!

[@hpongpong]

Great concise presentation. Thank you so much!

[@DevelopersDigest]

Thank you! 🙏

[@ryanroman6589]

this is super valuable. awesome vid!

[@DevelopersDigest]

Thank you! 🙏

[@rembautimes8808]

Thanks very nice tutorial

[@DevelopersDigest]

Thank you

[@brunozwietisch]

I’m looking to learn how to use Llama. Do you have a minimum configuration for the 8B version to run? Because here in Brazil, the dollar-to-real exchange rate is 6 to 1, and by the end of the month, the budget gets tight for those who want to learn.

[@DevelopersDigest]

Groq has a free tier for llama as well as cloudflare!

[@alejandrogallardo1414]

for models at ~70b, i am getting timeout issues using vanilla ollama. It works with the first pull/run, but times out when i need to reload model. Do you have any recommendations for persistently keeping the same model running?

[@DevelopersDigest]

github.com/ollama/ollama/pull/2146

[@rehanshaikh2708]

how can i use this endpoint in langchain chatollama?

[@nexuslux]

Can you use open web ui?

[@danielgannage8109]

This is very informative! Thanks :) Curious why you used a g4dn.xlarge GPU ($300/month) instead of a t3.medium CPU ($30/month)? I assumed the 8 Billion parameter model was out of reach with regular hardware. What max model size works with the g4dn.xlarge GPU? To put into perspective, I have a $4K macbook (16gb ram) that can really only run the large (150 million) or medium (100 million parameter) sized model, which i think the t3.medium CPU on AWS can only run the 50 million param (small model).

[@dylanv3044]

maybe a dumb question. how do you turn the stream data you received into readable sentences

[@DevelopersDigest]

You could accumulate tokens and split by the end of sentences . ! ? Etc and then send resp after grouping function like that

[@ConAim]

Stay away from AWS, it will cost you arms and legs in a long run..

[@DevelopersDigest]

Which vendors do you prefer? 🙂

[@BeCodeless-dot-net]

nice explaination

[@DevelopersDigest]

Thank you!