How to Run Llama 3 Locally on your Computer (Ollama, LM Studio)

🌟 Welcome to today's exciting tutorial where we dive into running Llama 3 completely locally on your computer! In this video, I'll guide you through the installation process using Ollama, LM Studio, and Jan AI, ensuring your data stays private while harnessing the power of AI. Whether you're a Mac, Windows, or Linux user, I've got you covered. Don't forget to hit the like button and subscribe for more AI-focused content. Let's jump right in!
👉 What you'll learn:
Downloading and installing Llama 3 on different operating systems.
Running Llama 3 using Ollama, LM Studio, and Jan AI.
Tips to optimise your local AI setup for speed and efficiency.
Real-time demonstrations and meal plan generation using Llama 3.
🔗 Useful Links:
Download Ollama: ollama.com/
LM Studio Website: lmstudio.ai
Jan AI Downloads: jan.ai
🔗 Resources:
Sponsor a Video: mer.vin/contact/
Do a Demo of Your Product: mer.vin/contact/
Patreon: / mervinpraison
Ko-fi: ko-fi.com/mervinpraison
Discord: / discord
Twitter / X : / mervinpraison
Code: mer.vin/2024/04/llama-3-run-l...
Jan AI Advanced: • Jan AI: Run Open Sourc...
📌 Timestamps:
0:00 - Introduction to Running Llama 3 Locally
0:24 - Starting the Installation Process
0:47 - Downloading Llama 3 via Olama
1:26 - Setting up Llama 3 with LM Studio
2:32 - Installing Llama 3 with Jan AI
3:12 - Using Olama API with Llama 3
3:55 - Running Local Servers with LM Studio
👍 If you find this video helpful, please like, share, and subscribe for more tech tutorials. Your support helps us bring more useful content to you!

[@JohnCorrUK]

Mervin ... My project for the week and you have just published this very helpful video 😊

[@sean_vikoren]

I am hoping to get this running in a vm. Hyper-V was not good at sharing the GPU, but VirtualBox is going fine so far, and the rumor is that it shares GPUs better. Good luck!

[@pratikkumar939]

U r brilliant. i was struggling and you came just in perfect timing

And that's why I expanded the GPU in Mac Studio! Fast-and-furious 🚀🔥👏

[@user-mk9tv3by4n]

very useful

[@MeinDeutschkurs]

Woohoo! This model will help me to place some decisions before processing the prompt. Not really agents, but a batch of to-dos. Typical function calling, I think. 🎉🎉🎉🎉🎉

[@nufh]

Which one is the best?

[@felipeclavijo1736]

YOU ARE INCREDIBLE !!!!!!!!!!!!!!!! do you know of any LLM running locally that can read and analyze data on my computer? for instance; being able to access excel files and to provide analysis within the laptop.

[@Mario_Kart_567]

When asking "Do you send data to Meta?" here is the LLAMA3 answer: "It's possible that some of the data I process or generate during our conversation might be stored on Meta's servers as part of their infrastructure. This could include: 1. Logs: Meta may store logs of our conversation, including the text we've exchanged, to help improve their platform and services. 2. Training data: The conversations I have with users like you can contribute to the training data used to improve my language models. 3. User behavior data: Meta might collect data on your interactions with their platform, such as search queries or clicks, which could be used to personalize your experience." You've been warned...

[@joseph-ianex]

Preprogrammed response, it doesn't send anything if run locally. You can run it without internet. Running on Meta's side or using their servers they are 100% taking your data.

[@jini611]

Mervin, thanks for the amazing video. Could you please create a video that elaborate the locally LLAMA 3 connecting to your SQL database and creates the sql query? I know you have the video that creates the SQL queries but I need to have a connection to that local database.

[@schmutz06]

just dabbling in this for the first time, when I ran that terminal command, where did it download llama 3? presumably to system32 folder which ollama defaulted to, but i dont see it. New to this.

[@user-wr4yl7tx3w]

can you do a video on Jan AI. not sure what it is exactly.

[@eduardocruzism]

How do I know if its using CPU or GPU? I mean, when I make a question my GPU usage goes from 1% to 30% and then to 1% again when its finished. But my CPU usage does the same. So is it using CPU or GPU?

[@FusionDeveloper]

Thanks, I didn't realize i could just open the command prompt to launch it. I assumed Ollama had it's own window and I was struggling to find where to open it.

[@magn8]

Same. I kept opening it.

[@jalam1001]

Thanks for video. I have been using llm studio. Its very slow. What's the hardware specification of your system?

[@dennissdigitaldump8619]

You absolutely have to have a GPU. The more VRAM the better. 12Gb is kinda the minimum.

[@stanTrX]

thanks but why do we have to download the model both for command line and for lm studio too.. aren't they the same model file? can't we use ollama serve??

[@anindabanik208]

Please make a video for local agent that run kaggle /colab using llama 3

[@firstlast493]

How about code competition in VSCode?

[@jets115]

Can you do a video on llama.cpp, api, and concurrent users?

[@eduardatonga7056]

Im new to this, is LM studio better than using Chainlit ?

[@jennilthiyam980]

is you approach totally safe for using sensitive data? is the model completely local or are you just using API method?

[@MervinPraison]

Save to use, as it’s running locally and don’t use api

[@secaja92]

Hi Mervin, could you tell me what are the specifications of your Mac? I recently ran LM Studio and noticed a spike in CPU usage after sending a prompt. I just want to confirm if this issue could be related to the specifications. my Mac is an M2 Pro with 16GB of RAM.

[@MervinPraison]

Yes it will spike , and I use M2 Max 32GB For normal model it’s working fine . But you can expect spike

[@themanavpaul]

No one would believe me, I ran it on my I5 8th gen U series CPU with 2gb Nvidia MX250. 1 query takes 50 mins to answer.

[@nhtna4706]

Pleas , Make a video to run Grok v1.5 locally , can u ?

[@JarppaGuru]

same as 2 and 1?

[@mikemartin8444]

Please answer this. I have an Nvidia 3090 (24gb) on a home brew pc. Can I run it on it? I just want to try running the models locally and don’t want to spend cloud dollars.

[@hardwalker95]

it should be alright for llama 3 8b. i read it requires 20gb of vram

[@negibamaxim9851]

i am duing that but insted of llama 3 i get the first llama

[@fiorellademedina8419]

Is this llama 70b or 30b?

[@quenlood70]

8b

[@Shaylenhira]

Is this free? Or does it cost you per API call you make?

[@joygumero]

How do I make Llama 3 recognize my mic and respond by voice?

[@eprd313]

I guess you'd have to install a good speech recognition as well as text to speech model and integrate the three. Or maybe you can find something like that already done in huggingface

[@sangu_akhirat]

Broo.. let me know about specification of computer, my computer run ollama soo slow. This is specification my comp: Intel Core I7 6700HQ CPU 2.60GHz, RAM 32 DDR4

[@etcgroup8811]

+10

[@Ginto_O]

does ollama use GPU?

[@quenlood70]

Yes

[@emanuelec2704]

When I use llama 3 8B on ollama or LM Studio, it is much dumber than on OpenRouter. Even after resetting all parameters to factory and loading the llama 3 preset. Even with the full non-quantized 8-bit version on LM studio.

[@fiorellademedina8419]

how do you know if it’s the version 8B or 70b?

[@emanuelec2704]

@@fiorellademedina8419 For the local model, it's at the beginning of the filename. And you can also tell by the size. For the OpenRouter version, it's always stated in the name of the model you are using.

[@socialtraffichq5067]

Hold on to your papers

[@podunkman2709]

Quality of ollama 3 is just hopeless. Just take a look at this ****: >>> How many liters of water per minute can a Dutch windmill pump out? That's an interesting question! The answer depends on the specific design and size of the windmill. However, I can give you some general information. Traditionally, Dutch windmills are designed to pump water from shallow sources, such as polders or wells, for irrigation purposes. The capacity of these windmills varies greatly, but a typical small to medium-sized windmill might be able to pump around 0.5 to 2 liters per minute (L/min). Some larger industrial-scale windpumps can pump much more, up to 10-20 L/min or even more, depending on the turbine design and the pressure head of the water. Really?

[@srenlarsen3148]

And still it cant do math. And still it do hallucinations And still it is old data from stop time = 2023 almost 1 year back and people still dont get it not up to date. But anyway sure it a bit better than Liama2 version. And still 70B is the online version to big to run local. Unly 7-8b in 2 and 3 version can be run local. But everyone has this so not really any new thing. In CMD, in Web UI, in Python, Javascript or what ever they what online platform they use. The unly real hype about it, is that it has some more data in the models. Else it is all the same thing. And yes free for everyone to use as they please them self like the older models.