Llama 3 3 on 64GB Mac Can it run?

[@sci1200]

Great video! BTW, what's the tool to display gpu/mem usage?

[@ComputerworxVideo]

Thanks, App is called mactop and can be Installed via Homebrew. Instruction and the code is here on github. github.com/context-labs/mactop

[@RachitDev]

I ran it on my 48 gb mac mini m4 pro… it works although really slow

[@MePeterNicholls]

I have M2 Ultra with 128gb ram. I tried the 70gb model. And it was pretty slow.

[@ComputerworxVideo]

yes about11 token per sec is not great but still usable. I ordered M4 Pro with 64GB i will test and update in comments her i do expect 4-5 Tokens per sec. How many GPUs u have and what is your performance?

[@Youtubeuseritis]

Intel Mac trashcan with 64gb ram and dual AMD d300 graphics card - good for llm? ​@@ComputerworxVideo

[@Youtubeuseritis]

Will it run on mac intel 64gb ram? Probably not?

[@ComputerworxVideo]

i would not use Intel mac due to Apple Silicon unified memory which works best for local llm. Also in a case of M1 and app cpu, ollama uses GPU. Where in case of intel it uses CPU. It will run but it will be painfully slow. I wonder if you even get 1 Token/sec.

[@bababababa]

but with what quantization?

[@ComputerworxVideo]

Hi, thanks for watching. I included a windows in video with all models and quantizations used. However the default 3.3 model called "llama 3.3 latest" its 70.6B Q4_K_M