LocalAI LLM Testing: How many 16GB 4060TI's does it take to run Llama 3 70B Q4

Answering some viewer questions and running Llama 3 70B Q4 K M with the 4060Ti's - how many does it take to run it?
Just a fun night in the lab, grab your favorite relaxation method and join in.
Our website: robotf.ai
Machine specs here: robotf.ai/Mach...
GPUs Being Tested:
4060ti 16GB amzn.to/3NeSEGT
GPU Bench Node
Open Air Case amzn.to/3U08Y27
30cm Gen 4 PCIe Extender amzn.to/3Unhclh
20cm Gen 4 PCIe Extender amzn.to/4eEiosA
1 TB NVME amzn.to/4gWFcFb
Corsair RM850x amzn.to/3NkITa4
128GB Lexar SSD amzn.to/3TZYYGh
G.SKILL Ripjaws V Series DDR 64GB Kit amzn.to/4dAZrWm
Core I9 9820x amzn.to/47UuIST
Nocuta NH-U12DX i4 CPU Cooler: amzn.to/3TZ7O6... CX299-PGF Logic Board amzn.to/3BxbWVr
Remote Power Switch amzn.to/3BubQOg
Recorded and best viewed in 4K
Your results may vary due to hardware, software, model used, context size, weather, wallet, and more!

[@dllsmartphone3214]

We need more channels like this that perform and showcase proper testing and hardware requirements for different models. Good job. I hope you will produce more related and useful content like this in the future. I wish your channel massive growth!

[@RoboTFAI]

Wow, I appreciate that! Just playing around in the lab and hoping people get something from it

[@jackflash6377]

Outstanding! So with a small investment a person could run a 70B model locally. In the past I have not been so happy with the Q4 size. It would be very interesting to see a comparo with the Q6 or Q8 model. Thanks for the time spent and the valuable info.

[@RoboTFAI]

Sure no problem we can do higher quants!

[@gaius100bc]

Ah, nice. That's what I was looking for. Not TPS, we already could predict that, but the combined power draw of 3 or 4 cards is what I was looking for. I expected power draw to be fairly low, but I was still surprised by all cards not pulling more than 200W at any given time during inference. Thanks for the test!

[@RoboTFAI]

Yea I concur, these 4060's sip power - and was a large reason I choose them for my specific needs originally. Still surprises me though.

[@luisff7030]

I made a test with LM Studio for only 1 GPU 4060TI 16GB: 100% CPU -> 1.38 tk/s (00/80 GPU OffLoad) CPU + GPU -> 2.00 tk/s (31/80 GPU OffLoad) 100% GPU -> 0.42 tk/s (80/80 GPU OffLoad) CPU Ryzen 9 7900 (105W power config), MSI RTX4060TI 16GB (Core 2895 MHz + VRAM 2237 MHz), DDR5 96GB 6000MT/s mradermacher Meta-Llama-3-70B-Instruct.Q4_K_M.gguf

[@aarond791]

Cooling has always been my biggest problem. My 2c about riser cables just so someone else doesn't run into the issues I had, if you intend on having more than 6 cards in a single machine, look into Retimer or at the least Redriver cards. Redrivers will effectively boost the signal which hopefully helps reduce signal loss. Retimers will re-package the signal and does a little more processing but ultimately was what I ended up getting. If you do this, its then possible to splurge on retimers with bifurcation built in, allowing you to extend your 16x slots into 8x8x slots if your board permits it, then you could in theory with a standard threadripper motherboard with 7*16x slots get up to 14 RTX 4090s into a single root. Training wouldn't be helpful, but if you were like me and wanted to run many models at the same time while also having enough GPUs to try super large models, this might be useful.

[@JimmyHacksThings]

You're a mad man! I've been working on a GPU node for my home k3s cluster, and getting into hosting a few jupyter containers. My hardware GPUs are mostly AMD 7900XTXs, but I look forward to testing out ollama and the 70B model on multiple GPUs. Also trying to do some locally hosted fine tuning. If all else fails, I can roast a few marshmallows. Hope the pizza was good!

[@RoboTFAI]

Sounds like you are also a mad man, prob with a good beard or mustache! If you are rocking those AMD cards - the viewers here would love to see some collab/results - lots of questions about AMD....

[@peterxyz3541]

How’s the performance from the 7900xtx? I’m looking into this as an option vs getting P40

[@JimmyHacksThings]

@@peterxyz3541 Sorry for the delay! The 7900XTX is on sale right now for ~$770 and does REALLY well for inference workloads. I was regularly approaching 100 tokens/s. The toolchain for anything else (like fine tuning) is completely in Nvidia's ballpark. So if all you're wanting to do is LLM inference, the XTX works VERY well. I'm quite pleased. (Running it with Ollama + Ubuntu 24.04).

[@sondrax]

I know it’s be SLOW… but very curious what 80B FP16 (Q16) would do? As that’s what we need even if we have to wait all night for an answer (but couldn’t wait 4 nights!). 🙃

[@miltostria]

I was watching the GPU utilization percentage and it seems that the average is around 25% for each GPU or am I wrong? Is it expected to be so or is there any configuration to utilize more GPU % during the inference?

[@RoboTFAI]

It's more like spreading the load, and using across the board VRAM rather than getting more speed or processing power. Also llama.cpp (under the hood here) isn't as great at multi-gpu setups as say Vllm or some others depending on your use cases/etc.

[@rhadiem]

It seems to me, looking at the output of the 4060's running here, that the 4060 is a bit too slow to be a productive experience for interactive work, and is better suited for automated processes where you're not waiting on the output. I see you have your 4060's for sale, would you agree on this? What is your take on the 4060 16gb at this point?

[@RoboTFAI]

I think the 4060 (16GB) is a great card - people will flame me for that but hey. It's absolutely useable for interactive work if you are not expecting lightning-fast responses. Though on small models the 4060 really flies for what it is and how much power they use. Lower, lower, lower your expectations until your goals are met..... I did sell some of my 4060's but only because I replaced them with A4500's in my main rig so they got rotated out. I kept 4 of them which most days are doing agentic things in automations/bots/etc while sipping power or in my kids gaming rigs.

[@fulldivemedia]

thanks for te great content, can you recommend any of these mobos for local ml and light gaming and content creation? msi meg x670e ace proart x670e rog strix x670e-e gaming wifi

[@rhadiem]

I would love to know what the cheapest GPU you'd need to use for a dedicated text to speech application running something like XTTSv2 for having your LLM's talking to you as quickly as possible. I imagine speed will be key here, but how much VRAM and what is fast enough? Inquiring minds... I mean we all want our own Iron Man homelab with JARVIS to talk to right?

[@RoboTFAI]

I don't work much with TTS, or STT - but we can go down that road and see where it takes us

[@JazekFTW]

Can the motherboard handle that much power consumption from the 3.3V and 5V from the pcie slots for the gpus without powered risers or extra pci-e power like other workstation motherboards?

[@InstaKane]

Can you test the phi-3.5 model? Would I be able to run it with 2 RTX 3090?

[@bt619x]

What are your thoughts/experiences on individual 4060 cards vs sets of 3090 cards with NVLink?

[@RoboTFAI]

Wish I had some thoughts on it. I don't have 3090's in the lab to test with (there may or may not be one in very near future).... the A4500's support NVlink but I haven't bothered to go down that route for inference and just let cuda/etc do it's job. I could make assumptions on speed/etc more so for loading, but they would just be that assumptions.

[@TazzSmk]

190W with 4 gpus seems pretty decent for performance it gives !

[@RoboTFAI]

I concur!

[@maxh96-yanz77]

Thx so much, your experiment show that 70B parameters just using 3x 16GB GPU is optimal in term cost/performance. Can we assume using 3x RX 6800XT 16GB (second use with cost about 150USD cheaper) more and less can handle 70B parameters ?

[@RoboTFAI]

I am sure they would handle it as far as offloading, can't say what kinda of speed of course. I am also not experienced with ROCm, or splitting on AMD cards but just to make a point I could dust off 24GB M40's from 2015 and run Llama 3 70b on them, or even run it purely on CPU with 48GB+ of ram....just wouldn't be quick at all. I can show you folks that if you really want!

[@maxh96-yanz77]

Thank u Mr. @@RoboTFAI it looks cool .. I use 1060 6G with RAM 64G, it works but token/sec was very bad. I am very confuse choosing either 4060 ti with 16G or Rx 6800 xt 16G or even RX 7900GRE 16G. Because I like to try LLMA3 70B moodel. with minimal cost , 3x RX 6800XT i think is affordable. The contraint choosing RX xxx , as i saw on tomshardware benchmark.. When you used for Stable Diffusing .. it's very-very sucks! .

[@fooboomoo]

I would be very curious how well those AMD cards will run

[@Viewable11]

AMD cards are one third slower than Nvidia for LLM inference.

[@JimmyHacksThings]

"For Science": Running llama3 7B on a single 24gb 7900XTX card with the prompt: "Make me a good Gobi Manchurian recipe." yielded: Prompt eval count: 22 Prompt eval rate: 1193.7 tokens/s Eval count: 783 tokens Eval duration: 8.3s Eval rate: 94.31 tokens/s. Server specs: Mobo: MSI MPG Z490 ATX CPU: i9-10850K RAM: 96gb OS: Ubuntu Server 22.04 - Integrated GPU used for console so the AMD is dedicated to ollama workloads Drive: 1TB NVMe Samsung SSD 970 EVO PSU: 1000W Corsair RM1000x I need to get a 4U case and a more capable motherboard to extend my GPU count.

[@pLop6912]

Good day you have on all 4060 all x16 lines PCIe, I would like to see a test with cut down to x8 lines and how it will affect the speed

[@RoboTFAI]

4060ti's only use/support 8x - and these tests are all running at 8x

[@IamSH1VA]

Can you also add *Stable Diffusion* to your tests?

[@RoboTFAI]

I don't have a ton of experience in the image/video generation side, but we absolutely could start doing those type tests and learn together

[@ecchichanf]

I use 7900XTX and 2 7600XT to run 70b Q4_K_M. I get 5-7 tokens per second but I still play with this setup. So RTX 3090/RTX4090 and 2x RTX 4060ti would be enough to run 70b. 2x fast 24GB cards like RTX3090/4090/7900XTX would be better for the speed.

[@moozoo2589]

The graphs are showing ~20-25% GPU usage each on all four GPUs. That also explaining 180W power draw in total, and not like 400W or so (100W per GPU). Could you please explain why it is not consuming all GPU power?

[@spookym0f0]

It implies it's limited by memory bandwidth. In theory you can squeeze more tokens/sec out of it by running multiple requests in parallel.

[@moozoo2589]

@@spookym0f0 I'm more inclined to think it is limited to single threaded CPU usage, resulting in tasks running sequentially on GPUs. Indeed, for some reason we don't see parallelism on CPU side, just one process at 100% usage.

[@milutinke]

Thank so you much for this

[@RoboTFAI]

Thanks for watching! I hope there is valuable information for the community, or at least some fun going on here

[@firsak]

Please, record your screen in 8k next time, i'd like to put my new microscope to good use.

[@yazi2879]

looks like he is running 4k with 100% scaling :(

[@GermanCodeMonkey]

Thank you very much for this 🙂

[@RoboTFAI]

My pleasure 😊

[@Phil-D83]

Intel arc a770 any good for this? (With zluda?)

[@RoboTFAI]

I don't have any to test with, but I do believe llama.cpp, etc, etc support Intel ARC with SYCL localai.io/features/gpu-acceleration/#intel-acceleration-sycl

[@koraycosar1979]

Thanks 👍

[@RoboTFAI]

Thank you too

[@Steamrick]

Hmm... I think rather than 3x 4060 Ti 16GB I'd prefer trying for 2x RTX 3090. Similar total price point, same overall memory and should be about 70% faster.

[@connmatthewk]

That's what I'm running and I'm extremely pleased with the results.

[@RoboTFAI]

Very plausible depending on needs, pocket book, power bill, etc - but prob exactly why people are watching this guy make questionable decisions with my money? All trying to figure it out

[@246rs246]

Do you know if the speed of DDR5 RAM on new motherboards is fast enough to partially store models on it?

[@RoboTFAI]

I run some models not fully offloaded on machines with only DDR4 in them - it's not fast by any means. I don't have any machines with DDR5 atm... expect my MBP - and that wouldn't be a fair comparison. We can test mixing CPU (RAM), and GPU (VRAM)

[@大支爺]

One 4090 + 192Gb DDR5 and you go.

[@RoboTFAI]

Yep more than enough for most things depending on your needs, and how deep your budget is.

[@BahamutCH]

Test with Exl2 format instead of GGUF. =) RTX for exllama, not for llama.cpp. =)

[@mjes911]

Still selling 4060s?

[@RoboTFAI]

still have a few - reach out [e-mail] or ping me on reddit/etc

[@TheMrDrMs]

hmmm to go 2x 3090 or 3x 4060Ti....

[@RoboTFAI]

checkout the newer video on the channel where I bring a 3090 into the lab, maybe will help inform ya?

[@iamnickdavis]

200w, crazy

[@RoboTFAI]

Agreed!

[@yaterifalimiliyoni9929]

Is the model actually giving coherent useful answer?? Accuracy should be tested. Whats the point of running a model thats inaccurate.

[@RoboTFAI]

Yep - Llama 3 is really pretty good. I don't focus on accuracy here (so far) as that is fairly subjective depending on what you are using specific models for and a tough subject to broach. I could attempt it but I would suggest folks like @matthew_berman (www.youtube.com/@matthew_berman) who I think does a really good job at comparing open source models when they get released.

[@yaterifalimiliyoni9929]

@@RoboTFAI thanks for the reply. I really appreciate the test your doing. I'm new and purchased my my 3060 laptop thinking i coukd run some models. Quickly realized not enough (vram) power or these models suck and thought running locally was hype. I think watching his channel is what lead to yours being on my time line.

[@perrymitchell7118]

405b

[@doityourself3293]

Do you have windows memory compression turned off.....????????????????? The biggest bottle neck in windows.

[@RoboTFAI]

I don't have windows machines in the lab, all linux based in the Kubernetes clusters