Run New Llama 3.1 on Your Computer Privately in 10 minutes

[@SkillLeapAI]

Check out our updated course on running private AI chatbots on your computer. bit.ly/skillleap

[@mal-avcisi9783]

Are you serious ? Using MacOS ? No one is using MacOS. Stop making videos showing howtos with MacOS.

[@joeldowner2991]

try AIJoel - Multi Generator: text, code, image (create sticker, remove image background, add color to black white image, image to video, logo design) and (music and video are in beta mode)

[@IndieGamesSpotlight]

so I installed ollama and asked it, are you cloud based or are you running locacly on my computer? It replied that is cloud based Ai. Why is that?

[@Alvin-i2t7o]

One new SSD and a full install later.... It works!! Docker was giving me an error which I couldn't resolve but it was all straightforward on a clean installation!

[@AliHassan-wc6nb]

Any luck?

[@mr.cannon]

PC USERS IF YOU ARE GETTING THE WSL ERROR WHEN RUNNING DOCKER - Enable virtualization in BIOS: This process varies depending on your computer manufacturer and model. Generally, you'll need to restart your computer and enter the BIOS settings (often by pressing F2, F10, or Del during startup), then look for an option related to virtualization or VT-x and enable it.

[@joeduffy52]

I get this error but can't see anything in BIOS like you mention. My M/B is the Gigabyte X570 Aorus Elite.

[@mr.cannon]

@@joeduffy52 Enter the bios in advanced mode - go over to Tweaker tab - go down to advance CPU settings - and enable SVM mode

[@faridgasimov1742]

Just returned to leave this comment

[@marcusstone6273]

Hey bro I just want to say that you grind is on another level. So good that you can go so for many years and sitll create new channels and succeed. Nice transition to AI content and your views are great too. Hope you get a lot of sponsorship and affiliate deals.

[@SkillLeapAI]

I appreciate that!

[@muhammadasad8549]

Brilliant. I have been looking for this videos since meta announced 3.1. Hats off.

[@muhammadasad8549]

@SkillLeapAI I should be really thankful if u can make a video to deploy it on server.

[@Hilmz]

Not private, its a hybrid model, caches data when offline then when connected back you can see it sends data. Use wireshark it will show you its sending data

[@longboardfella5306]

This is an important comment because many channels are saying OpenWebUI and Llama3 is private to your machine. Is there any way to turn off the cache sending process? I would like to analyse documents that are not permitted to be publicly uploaded

[@MihaMartini]

@@longboardfella5306 Ollama and Llama3 are private, but OpenWebUI might send some analytics and other data.

[@bryanjuho]

Is this true? Any resource that supports this statement?

[@DudethatGross]

@@longboardfella5306 run it in a VM or container that has the network adapter disconnected from the internet, or wifi off entirely

[@andresshamis4348]

Llama is literally private, maybe openwebui…. However what would the ui need to cache and send over the internet? Doesnt make sense to me

[@tikkivolta2854]

the really interesting part around 07:15 is that in a few months computing power and size of these models will make it possible to run them on practically anything. when they get more effcient we'll have them in our phones.

[@SantiagoAbud]

That's the future of this technology. Not that I endorse it or judge it in any way, but it's the way the development is heading.

[@thinhngo7244]

OpenELM from Apple is already runable on mobile devices I think

[@tikkivolta2854]

@@thinhngo7244 i ran LLAMA 3.1 8b on my laptop with docker and ollama. like a breeze.

[@skywalkerjedi95]

Thanks! This video was awesome and really detailed! Can’t wait to start trying this.

[@Ilan-Aviv]

great video for local ai. easy and clear explanation. thank you!

[@iskandarhussain]

Perfect ‼️ Just the video I was looking for with intro to how to upload files to the model‼️ THX‼️

[@GmanBB]

You have great teaching skills. Thank you for making it so simple!

[@pertsiya]

Thank you for your will to share with us!

[@womble_1034]

subscription incoming!! great content, keep up the good work

[@FastWReX]

No joke. I've always hated Docker. Hated everything about it. However, seeing you run the Openweb UI command and it just randomly showed up in the Docker app is making me reconsider. Holy moly! Might have to reinstall it on my Raspberry Pi 5.

[@southcoastinventors6583]

Sounds like step one is buy a $5,000 M3 if you want to run it locally now at least before they release smaller quantizations of Llama 70B and 8B

[@melaronvalkorith1301]

Llama3.1 8B runs much faster than you can read on an RTX 2060 Super. Not dirt cheap, but I built my PC for $1.4K 4 years ago - should be cheaper now, and I built it for gaming, not AI. You can host it on a desktop and connect your other devices with a VPN like Tailscale. Don’t spend extra money for less performance by going for small form-factor/laptop.

[@HitsInSandbox]

Go grab one of the older servers that can be had for $100 - $150 with the dual XEON 10 core cpu's and boost it up from the 64gig to 128gig running 20 cores it runs well but you won't be able to stick a modern video card in it. But it still does a good job for a system that can be built for $200 Much better than 3 i10 systems running. Also warning get ready for a bigger Hydro bill though.

[@CodyAvant]

I run 8b on my 2020 M1 MacBook Air (8gb ram and 7 core GPU) and the output token rate is much faster than a normal speech cadence.

[@mcombatti]

I have a 9 year old windows computer that runs 13b and 8b models the same speed as your brand new $5000 Mac 👀 it was purchased at a Walmart for $270 in 2015 😂

[@bassamel-ashkar4005]

Bullshit detected ​@@mcombatti

[@b34k97]

"We need to go to an app that a lot of people have never used before.... its called 'Terminal'". OMG that line had me dying!

[@SkillLeapAI]

I’ve made videos for 8 years and 99% of people never used terminal

[@b34k97]

@@SkillLeapAI No I understand, and it makes perfect sense. Just as someone who's used a terminal at school and at work for the past decade... the delivery of line just tickled my funny bone

[@ZavierBanerjea]

Absolutely hilarious...

[@danielrossy7453]

Same dude : ) And then my wife came into the room and said "are you really?" HAHAHA

[@kamaboko1]

Awesome! Thanks for the post!!!

[@micbab-vg2mu]

Great - I will try 70B - ) thanks for instruction how to do it:)

[@burntdoug6823]

What's your PC specs that can handle 70b?

[@Mewmew-y4m]

@@burntdoug6823 Cpu: intel i5 2400 Ram: 4gb Gpu: integrated Graphics Storage: 120Gb Hhd

[@PowerON-Tech]

Thank you very much for this video.

[@kevinmiole]

I hope they give access to internet. Thank you for this

[@wettingfairy6764]

讲的很清楚，很棒的入门指引

[@hydron7150]

wanted to try 70b with 4070ti 12gb ryzen 5 7600x 64gb 6000mhz ram and it is pretty slow, it takes 20 seconds to response "hi" prompt 😄

[@tikkivolta2854]

i would love for you to create a tutorial how to train these models on specific data. any chance?

[@SkillLeapAI]

Adding to the list

[@nohandle8008]

@@SkillLeapAI awesome, thank you. I have a specific use case requiring very specific data to train the model, would love to see how effective it can be. Also concerned with the data flowing back online, can you elaborate on what is sending date back out when the machine is reconnected to the internet as others have mentioned?

[@digigoliath]

I do appreciate this informative walkthrough though! TQVM!!

[@vladyslavklochan4181]

Thank you for tutorial.

[@Kevin-fp5zo]

Hello! What kind of hardware setup do you have to have to run OpenSource LLMs? LLMs are actually quite heavy and they require high GPU, RAM, and CPU power. Can you do a KZbin video about which computer power parameters or PC brands are optimal for running them smoothly? I love your content. Keep up the good work! :)

[@betabishop3144]

They are indeed quite heavy but in case you haven't noticed, in each of the LLMs documentation there should be a section dedicated to the hardware requirements, google has it, meta has it and the other probably do too. You could look them up like "Llama 3.1 hardware requirements" and the first link should take you there

[@yetkindev]

you are the perfect man thank you

[@fangeming1]

How much vram is needed to run the model depends if the model of quantised or not. This should be explained in this video instead of giving contradictory information.

[@tikkivolta2854]

as much as you are correct i am fairly certain "giving contradicting information" wasn't the intent.

[@moonduckmaximus6404]

Hey do you know where we can get a comprehensive explanation to what we downloaded? i cant afford his course

[@naeemulhoque1777]

Nice, straight forward video. Can you make a Hardware buying guide please?

[@II-qh7xn]

worked with issues hats off

[@MrAshwin27]

Huge respect

[@null4624]

thanks dude, I was able to run llama3.1 8b on a linux laptop with 8gb ram and am impressed..

[@shampaghosh1241]

Wow did you do any special modifications? My device is also has 8GB ram and an Intel i3 processor do you think I can possibly run it with decent speed?

[@null4624]

@@shampaghosh1241 No special mods, just selected the smallest model and followed the steps from this video and ran some prompts.

[@HitsInSandbox]

The 405B version needs at least 768Gig of Ram as it uncompresses the 200+ gig in memory and runs uncompressed vector links from memory to run effectively. But should you do it it beats OpenAi hands down. Fully uncompressed it could be from 2 to 4 terabytes in size.

[@gaathastory]

Reminds me of last two seasons of TV series Person of Interest…. Store “AI” on massive RAM capacity sticks

[@yetkindev]

you have a great internet :D

[@mikemaldanado6015]

Nice video. I dont like u have to sign up for webui and don't like docker for security reasons but your fist two steps helped alot.. btw you can upload or pass large files on the command line. finally microisoft llama does not meet the traditional definition of open source, because it's not. What they did is create a new definition, their definition, and put it in the terms of service..... must be nice to be able to change the def of words willy nilly. also nothing we have today meets the official comp sci definition of AI, not by a long shot.

[@rafaeel731]

Thanks for the vid, a couple of confusions to share: 12:18 how can a LLama 3.1 model not know anything about LLama 3 because of delay? it doesn't make a lot of sense Plus you compared 8B on a text exchange while you gave the 70B model a python code to decipher, then you gave the 8b a text file to summarise. We can't compare execution times unless the task is identical

[@puccaso]

9:20 i believe that there is also a docker credentials package that also works, and doesn't require the GUI bloat.

Perfect. Thanx

[@mediatechtube]

Nice video. What are the use cases for running AT locally on your computer at home? Whats the purpose when people can get a subscription? I can think of a few but i would like to know what others think. Cheers!

[@MoonyongKim]

Hi. First of all, thanks for the video. It's really useful and easy to follow step by step. I am running M1 mac air and seems like it's not good enough to run llama 3.1 as it seems to freeze my computer. Which model would you recommend for M1 mac air?

[@blackgptinfo]

Great video. Did you test how many documents you can upload at once and have it summarized?

[@SkillLeapAI]

I haven’t yet but it think it’s a good amount

[@sabuein]

Thank you.

[@andyli541]

Is there a way to bring this local running Llama 3.1 onto my website? I want to share my trained AI with other people. Thanks!

[@bilza2023]

There are special server on digital ocean .. but simply you installon your website and make it available throught an API.

[@robwin0072]

Great video. Question: can I redirect all models downloaded (installed) in Ollama, to a secondary drive inside my laptop? C: Primary System M.2 SSD 2T D: Secondary SSD 2T

[@walter3663]

Thanks for the great tutorial. Can you let me know the defaut path to store chat histories? Is it possible to change it?

[@NVX_Ink]

What would be an affordable, yet ideal, desktop workstation

[@terrysh7264]

Hi. TY for this video. I'm wondering, do I need to train the model that I install?

[@SkillLeapAI]

No you can just use it after install

[@ArekMateusiak]

Hi, does anyone knows what is needed to run well full 70B Llama 3.1 model? so it responds quickly?

[@RiftWarth]

Wow the 405B model file size is still smaller than some of the Call of Duty games. LOL

[@SkillLeapAI]

Yea that’s true but it’s basically text. If it was video, it would a million times bigger

[@HitsInSandbox]

It is layer compressed and thanks to Ai is the tightest compression of any type of computer files to exist is AI. Just because it is 250G that is compressed as fully uncompressed it might be more like 2 to 4 terabytes and still using a pointer index for all the words in all the languages it supports. The same compression would take a Blueray movie of 4.5Gig and compress it down to like 50 or 60 meg space. But the layout of data is totally different for different needs.

[@hiteshdesai2152]

this is great, thanks for puting in such simple and understandable way. I can run locally now, is there a way where I can point it out this local models to my python code, or my langchain/llama_index application code?

[@karthikb.s.k.4486]

Inference looks fast in local what configuration of Mac laptop are using please let me know. Thank you for nice tutorial.

[@SkillLeapAI]

It’s the top of the line m3 Mac with 64 gig of ram

[@gRosh08]

Crazy cool.

[@gaganmadhan733]

Can we store the files on cloud storages like AWS S3 and then run it or deploy it ??

[@Kingkimabdu4090]

Everything seemed fine until I clicked the link in Docker. The website page opened with an error message stating, "This page isn’t working." Can anyone offer assistance?

[@arnolda7150]

Than you so much. Can you tell me how to acces openAi with WiFi on or off?

[@kick_kisu]

Great video. How can I run the docker for open webui on my Linux local server, but ollama running on my windows pc?

[@HitsInSandbox]

You can use msty without docker

[@NeoDon1]

Those specs for the 405b are not right. I have 64gb's ram and mine flies.with 4080 super and AMD 5900x

[@piska4f]

Well, if you try to use the same token length that's suggested in their website, it would require much more computing power than what you already have... it's set to 2048 by default...

[@Eldorado66]

Well it's probably walking, not flying.

[@HolographicKode]

What's your hardware setup you run this model on?

[@SkillLeapAI]

I’m on m3 Mac 64 gig ram. I can run the 70B model and the small models respond almost instantly

[@HolographicKode]

@@SkillLeapAI mac book pro? mac pro? how much VRAM? (complete configuration). this has to be a $5K+ setup i suspect.

[@frankvasquez4827]

If I installed the 8B model first and then I want to install the 70b, Will I have both installed or the largest one will overwrite the 8B? Can I uninstall the models, just to save up some storage space 😅 (asking because I'm not too technical about it, I'm using Windows btw). Thanks in advance.

[@SkillLeapAI]

It will install both and you can choose between them. It won’t override. And yes you can remove. My new video covers that and some new upgrades

[@frankvasquez4827]

@@SkillLeapAI Thank you, I will watch it. I managed to install them with this video!

[@0reo2]

I see chatgpt is becoming the edge of LLM: hey Chatgpt, how do i install this other LLM i want to have?

[@gRAVItation1988]

Great job! I have a M1 mac with 16gb ram. Can i run 8b?

[@SkillLeapAI]

I think so

[@tikkivolta2854]

​@@uncannyrobotdo you also train models and would care to elaborate? i'd be all ears

[@tikkivolta2854]

@@uncannyrobot i will find one, thank you!

[@naveenkumarmurugan1962]

lovely

[@matrixpredator]

you can run 70b models with 30gb ram.... and for 8b models you dont need 32. ofc i mean quantized models

[@andreac7389]

Hi, is this model multimodal like ChatGPT 4 Omni? I mean, can it generate code, solve mathematical problems, etc., or is it purely a linguistic model capable of easy conversation but unable to handle complex issues? In other words, my question is, do only the models hosted on the servers of OpenAI, Anthropic, or Meta have the capability to manage complex problems, or does this offline model also have that capability? Thank you.

[@SkillLeapAI]

They have some of that capability but online models are going to much better. It’s very difficult to run the better models offline. The best version of llama 3.1 is too complex to run on a computer and OpenAI and Anthropic dont have an open source model that you can run locally

[@thevoice6853]

can you do a tutorial on how to do it on windows, thanks

[@billl1715]

The steps doesn't show how the models gets loaded onto WebUI. When I'm in WebUI, there are no models.

[@SkillLeapAI]

They should automatically. I didn’t do anything in webui to add them. They were added through terminal in easier steps

[@Xiaoklunar]

On the top corner there is choose model

[@zunairakhalid7358]

Can we do a APi call of this local LLM in my code ?

[@frankbaron1608]

terminal is called "cmd" on windows.

[@christerjohanzzon]

So, you don't need a fancy GPU to run Llama locally? It does say that you need an Nvidia GPU...but you're running a Mac? Please elaborate.

[@SkillLeapAI]

I have the built in Apple GPU. These are my specs. Chipset Model: Apple M3 Max Type: GPU Bus: Built-In Total Number of Cores: 40

[@SkillLeapAI]

The 8B model should run on variety of GPUs

[@christerjohanzzon]

@@SkillLeapAI Ah, I see. Thanks for explaining. :)

[@longboardfella5306]

I believe modern Macs run a unified memory model which combines and distributes the GPU and CPU memory as needed. PCs don’t do this so need a specific amount of GPU ram on dedicated Nvidia GPUs to run models. I have an RTX8000 which is 24GB VRAM which runs all 8B models fine - but completely chokes on 70B models regardless of quantisation. For Macs it’s all about your total memory available and having enough modern GPU cores to do the CUDA processing as I understand it

[@filipskerik1477]

The Docker "thing" is only for embedding the model installed before to the webUI? Or its something that pushes all of the stuff to cloud and I don't need that NASA computer? Thanks

[@sairajulu4395]

I gave prompt to give a word without vowels, lamma3 gave START then I said its wrong then final output is STRT, still it required more training lol,

[@WIWUPRODUCTIONS]

My docker won't show the address/ports after installing everything. What can be the problem? I'm trying to google it all day and can't find an answer

[@Caged_Monuments-x6p]

So I got it working and My CPU is running 75% and I have a 16 core amd 3950x. My GPU is barely running.

[@SkillLeapAI]

Nice

[@Repz98]

*Requirement* at 7:56

[@Repz98]

Get 25 of "AMD Radeon RX 7600 XT 16GB". When not using the Llama go back to mine crypto.

[@extremelylucky999]

Would like to learn to do Llama + Groq + iPhone shortcuts to run llama.

[@ndidiahiakwo7412]

Will the website version be capabe of uploading documents anytime soon? My computer isn't large enough to support running the offline models.

[@SkillLeapAI]

Not sure. I hope so

[@moonduckmaximus6404]

I can try to help you find information on the web, but I'm a large language model, I don't have direct access to the internet..

[@sumitksrivastava]

Why do I get 400: 'NoneType' object has no attribute 'encode' error anytime i try to upload a document?

[@ai_snillet]

When I try to run open-web ui, Flowise opens up. I guess they are listening to the same ports =/ ? Can I solve this in any way?

[@jeremy4510]

Can you do a video like this for using llama in Python

[@jitendravishwakarma7949]

gemma2:2b is insane model compared to its size

[@nessim.liamani]

Can we locally remove restraints on LLaMA models, including ethical safeguards? Thanks

[@moonduckmaximus6404]

Thank you for your effort. Any reason why my LLAMA3.1 would respond one letter @ a time? 70B

[@HitsInSandbox]

YOu need more memory and likely a AMD or Nvidea video card with at least 8 gig ram 12 to 16 is better. But the 70B needs at least 64Gigs of ram not 32 or less. Otherwise your just using Virtual ram which will eventually crash it as it chokes on larger inputs. If you have the specs then your Anti Virus is likely bogging it down checking all data movement in memory.

[@moonduckmaximus6404]

@@HitsInSandbox 4090 with 128 gigs of ram....i dont use an antivirus

[@NeptuneGadgetBR]

Hi, I couldn't run docker on my GPU, I have RTX 4090 which should help a lot, while on CPU is slow, do you have any idea how to enable my GPU on Docker - Windows 11?

[@fl028]

Use --gpu all Option :)

[@moonduckmaximus6404]

hey i keep getting a notification that llama 3.5 is ready to update from llama3.1? i click the notification in windows 11 but nothing happens. how can i verify or update llama 3.1?

[@volebien]

thanks a lot. That's what i wanted to do. Now i can upload files and not pay for chatgpt plus. But anyway, it is very slow and uses a lot of cpu. Do you know any tweaks where i can share the workload with the gpu?

[@SkillLeapAI]

Only to use smaller models. Not llama. But like phi 3 or the smaller ones

[@SimonFeay]

when I go to workspace I don't seem to have the "Documents" tab.

[@nosuchthing8]

Wait, what does docker do if the LLM is running?

[@dawnbunty7]

i have macbook m3 pro with 18 gb ram and 500 gb harddisk will this be sufficient

[@lucifergaming9491]

i use ubuntu my web ui doesnt show any models after correct installation

[@IbrahimAli-hf7mq]

+1

[@swarnimdubey]

How much total of the storage it requires anyway?

[@Carlzora]

Are you able to upload images to use with prompts?

[@SkillLeapAI]

Most open models don’t have vision and if they do, they are not good. I would use ChatGPT for that

[@longboardfella5306]

Lava models are pretty good for image analysis

[@mohdalki7271]

Can I train my data?

[@SelvamuthuMR]

hugging face llama 3.1 model repo storage is 60 gb but it run very slow for one response and ollama run same llama 3.1model faster but size of llama 3.1 is around 5 gb. what is the difference

[@prajwalm.s7976]

Can I fine tune the 70B model and use the open web ui?

[@HaraldBendschneider]

I downloaded the Windows file "OllamaSetup.exe" and installed Ollama. What now? After clicking on the app-icon nothing happenes. Is there any tutorial for Windows out there? Running the app in CMD, I cannot use the shortcuts: C:\Users\user\AppData\Local\Programs\Ollama>show The command "show" is either misspelled or could not be found.

[@fotszyrzk79]

Hey! Open cmd once again and type "ollama run llama3.1" you can play with it in command window. I'm looking for interface now, to run and play it in a nicer way - docker mentioned by the OP need subscription - could be 0 USD, but I don't like to subscribe.

[@fotszyrzk79]

If you type just ollama it will print the available commands for you.

[@HaraldBendschneider]

@@fotszyrzk79 Thank you! This worked. I can chat in the command window. But I wanted to have an UI and I don't understand what the icon in the taskbar window is for. View logs and Quit Ollama is all I can do.

[@longboardfella5306]

@@HaraldBendschneiderOpenWebUI gives you the windows chat interface. I use it on Windows 11 and it works fine with Ollama and Docker. You can look at Matthew Berman’s channel for simple install instructions as well. Bottom line: most tutorials assume Mac but it’s not hard to get it all working on Windows - you have to hunt a bit more for work arounds

[@DenofLore]

1.2 tokens a second at fp16 on a 4090 and 14900k. 😒

[@SkillLeapAI]

wow that is slow

[@DenofLore]

@@SkillLeapAI and that’s the 8B model 🙄

[@ronilevarez901]

@@SkillLeapAI Slow is 2 seconds per token with no previous context using GPT2, which is the only model my pc can load. You guys are being given the world on a silver plate and you still call it slow 😑

[@HitsInSandbox]

Go with the Q8 version and work up or down from their. Remember the Graphics CPU still takes orders from the much slower computer CPU and memory on the MB. If you only have 16GB Ram and a 16GB Nvidia the computer will slow the info flow going to to the video card which will always be waiting for more data which it can not pass fast enough to the RISC based video card. Think of it like a 64bit computer trying to send data to a multi core Nvidea RISC 128/256bit CPU system. The video card is slowed down waiting by as much as 400% for the slow computer CPU more so if it is an Intel which is not RISC designed for information flow. That is where AMD tends to leap ahead of Intel by being designed around the RISC architect. The Intel 64Bit CPU is having to convert and piece together four packets of info before the video card can recognize the much slower computer CPU. Many people today don't realize that Graphic CPU's have left regular computer CPU's in the dust for speed, performance and pricing many years ago. There are RISC graphics CPU's that can be had for around $8 that outperform i10 Intel chips going for around $360. Only problem is the boards to run them are not openly available to the consumers.

[@DenofLore]

@@HitsInSandbox 64gb 6800 and a 24gb nvidia BAR enabled. Z790 aorus master. Mistral’s new 13B with only partial offload (23 layers) gets 9-14 same quant and size. It’s an Ollama-side inference issue. What that is I have no idea I just implement the things and can narrow down based on experience. Yes different models but unless Meta is just bad at that or the model has inherent training and architecture that makes inference difficult to optimize it’s just this generation of engines build that isn’t there yet to handle it. Give it a few updates it’ll probably improve but right now it’s dogshit.

[@CodeCraftHub-NAS]

could you put the file on your web server then use it to be your search and or help?

[@JarppaGuru]

8:41 no need docter blabla bla. we just unzip on windows hit bat