Zero to Hero LLMs with M3 Max BEAST

M3 Max is a Machine Learning BEAST. So I took it for a spin with some LLM's running locally.
I also show how to gguf quantizations with llama.cpp
Temperature/fan on your Mac: www.tunabellysoftware.com/tgp... (affiliate link)
Run Windows on a Mac: prf.hn/click/camref:1100libNI (affiliate)
Use COUPON: ZISKIND10
🛒 Gear Links 🛒
* 🍏💥 New MacBook Air M1 Deal: amzn.to/3S59ID8
* 💻🔄 Renewed MacBook Air M1 Deal: amzn.to/45K1Gmk
* 🎧⚡ Great 40Gbps T4 enclosure: amzn.to/3JNwBGW
* 🛠️🚀 My nvme ssd: amzn.to/3YLEySo
* 📦🎮 My gear: www.amazon.com/shop/alexziskind
🎥 Related Videos 🎥
* 🌗 RAM torture test on Mac - • TRUTH about RAM vs SSD...
* 🛠️ Set up Conda on Mac - • python environment set...
* 👨‍💻 15" MacBook Air | developer's dream - • 15" MacBook Air | deve...
* 🤖 INSANE Machine Learning on Neural Engine - • INSANE Machine Learnin...
* 💻 M2 MacBook Air and temps - • Why SILVER is FASTER
* 💰 This is what spending more on a MacBook Pro gets you - • Spend MORE on a MacBoo...
* 🛠️ Developer productivity Playlist - • Developer Productivity
🔗 AI for Coding Playlist: 📚 - • AI
Timestamps
00:00 Intro
00:40 Build from scratch - manual
09:44 Bonus script - automated
11:21 LM Studio - one handed
Repo
github.com/ggerganov/llama.cpp/
Commands
//assuming you already have a conda environment set up, and dev tools installed (see videos above for instructions)
Part 1 - manual
brew install git-lfs
git lfs install
git clone github.com/ggerganov/llama.cpp
cd llama.cpp
pip install -r requirements.txt
make
git clone huggingface.co/teknium/OpenHe... openhermes-7b-v2.5
mv openhermes-7b-v2.5 models/
python3 convert.py ./models/openhermes-7b-v2.5 --outfile ./models/openhermes-7b-v2.5/ggml-model-f16.gguf --outtype f16
./quantize ./models/openhermes-7b-v2.5/ggml-model-f16.gguf ./models/openhermes-7b-v2.5/ggml-model-q8_0.gguf q8_0
./quantize ./models/openhermes-7b-v2.5/ggml-model-f16.gguf ./models/openhermes-7b-v2.5/ggml-model-q4_k.gguf q4_k
./batched-bench ./models/openhermes-7b-v2.5/ggml-model-f16.gguf 4096 0 99 0 2048 128,512 1,2,3,4
./server -m models/openhermes-7b-v2.5/ggml-model-q4_k.gguf --port 8888 --host 0.0.0.0 --ctx-size 10240 --parallel 4 -ngl 99 -n 512
Part 2 - auto
bash -c "$(curl -s ggml.ai/server-llm.sh)"
💻 MacBooks in this video
M2 Max 16" MacBook Pro 64GB/2TB
- - - - - - - - -
❤️ SUBSCRIBE TO MY KZbin CHANNEL 📺
Click here to subscribe: / @azisk
- - - - - - - - -
Join this channel to get access to perks:
/ @azisk
#m3max #macbook #macbookpro
- - - - - - - - -
📱 ALEX ON X: / digitalix

[@AZisk]

JOIN: youtube.com/@azisk/join

[@AdamsTaiwan]

Just tried the LM Studio and my desktop. Was able to connect my 8 year old notebook's vsCode using Code GPT to it. Pretty nice, but still looking for a solution that can scan all my vs solution and tell me where to fix my problems.

[@MaxTechOfficial]

Keep up the good hustle, Alex! -Vadim

[@AZisk]

Thanks Vadim!

[@univera1111]

@@AZisk if I may ask, can you replicate this on a Linux or windows and see which is easier for users. Or u can just say here

[@zt9233]

@@univera1111also benchmarks

[@abhishekjha9041]

​@@AZisksir please make a video for MacBook pro specifications for Machine learnings . I'm so confused about what to buy 16inch with 30 core 96gb ram Or 16inch with 40 core 64 GB ram. Or I have to buy a m3 pro 18 core 36gb ram. I'm so confused and like me other people also so please make a separate video on that it's a request

[@abhishekjha9041]

​@@AZiskAnd I have a question that I do some research and find out that MacBook pro in Delaware have zero sales tax which means if I buy MacBook pro in 2500 dollars so I don't have to give any tax on it. It's is true sir.

[@Kevin-hx3ci]

Alex I am so happy I found your videos on KZbin because I had been looking for someone to help tutor me on tech stuff on Mac. Can’t express how helpful this has been for me.

[@giovannimazzocco499]

Excellent stuff. I searched KZbin for weeks to find benchmarks of DNN models on M3. This is the first and only one I've found so far. There's is a ton of videos on video editing, graphics, gaming and music production on M3s. But for what concerns fresh material about machine learning on Apple Silicon I'm pretty convinced you're the only game in town. Keep it up. Looking forward to seeing more benchmarks.

[@atldeadhead]

I enjoy all your videos but this one was particularly interesting. I look forward to future videos that explore machine learning leveraging the power of the M3 Max. Fantastic stuff, Alex. Thank you!

[@catarinamoreira4805]

This is fantastic! Thank you so much! More content on LLMs, please!

[@anthonyzheng7274]

You are awesome! This is great, I bought an M3 Max several days ago and really having a great time playing around with LLM's.

[@someone5781]

So excited for your next video on training on the m3 max!

[@ismatsamadov]

I subscribed a few months ago, but I have never seen such quality content. Thanks, Alex! Keep going.

[@AZisk]

thx 🙏

[@facepalmmute3619]

the bass in your voice on the MBP speakers is phenomenal

[@SebastianWerner82]

Great to see you creating videos with this type of content as well.

[@JohnSmith762A11B]

Excellent. Many thanks for putting this together! 🥂

[@suburbanflyer]

Thanks for this Alex! Just got an M3 Max so it'll be great to try out some new things on it, this definitely looks interesting!

[@bawbee27]

Incredibly helpful - this is the video everyone with an Apple Silicon machine trying to do LLM’s should see!

[@juangarcia-wp2zr]

very cool content, thanks, I feel very curious now to try out some of this llms

[@nikolamar]

Alex this is AWESOME!!! Thank you!

[@mr.w7803]

Dang!! Dude, this video sold me on that M3 Max configuration… this is EXACTLY what I want to do on my machine

[@tonbii]

i bought M1 Max with 64GB 3 years ago to do this kind of works. I am so happy to find this video.

[@joshgarzaBI]

Awesome video here. I'm bummed I didn't do it sooner. I have never seen my M1 (16GB) freeze before. Great teaching here!

[@scosee2u]

I really love your videos and how you explain these cutting edge concepts! Would you consider researching or interviewing someone to make a video about quantizing options and how it impacts using llms for coding? Thanks again for all you do!

[@AZisk]

Possibly!

[@RadAlzyoud]

Brilliant. Thanks for sharing.

[@bdarla]

Super helpful! I hope you will continue with further relevant videos!

[@estebanguillen8110]

Great video, looking forward to the LLM fine-tuning video.

[@JasonHorsnell]

Just got myself an M3 Max and found your videos. You’ve saved me SO MUCH TIME….. Very much appreciated…..

[@danieljohnmorris]

How much ram?

[@JasonHorsnell]

⁠36GB max base. More than enough for my purposes atm.

[@TimHulse]

Same here!

[@_mansoor]

Awesome, Thank you. Halo Alex!!!🎉🎉

[@jorgeluengo9774]

Thank You Alex, this is an amazing video. I will look into the software development tools installation.

[@AZisk]

Awesome! Thanks

[@LukeBarousse]

Interesting, I didn't know about LM Studio; that makes things A LOT cleaner

[@aimademerich]

Thank you for the GPU setting in LM Studio at 15:00!! Can you do more videos on proper GPU setup on LLM's for M1-3?

[@dennisBZC]

Hey Alex, I’ve been watching many of your videos, mostly for comedy - as I find you hilarious the way you explain things to a non-tech mortal, but occasionally, try to copy your instructions and try my luck to test out a few things for fun. I’m not one for cutting code, but I still watched the whole thing, just to get to the LM Studio to download a model to try out on my M3 Max. I tried the Phi3, thinking Microsoft might be better than the others. I don’t have a clue what I’m doing, but it seems to work a little. You are a LEGEND! Keep up the great work. Love to see how you train your AI in due course. I keep shouting at it to “sit”…my MacBook hasn’t moved, so I guess, it is quite obedient.

[@MikeBtraveling]

very interested in the topic and would love to see you do more in this space.

[@XNaos]

Finally, I waited for this

[@geog8964]

Thanks, Alex.

[@jameshancock]

Nice! Thanks! FYI when you change the preset you’re changing how it inputs into the LLm. Which caused it to go nuts.

[@devdeal4146]

Just got the m3 max with 48gb ram. Excited to see how it works with your tutorial. Thanks!

[@theperfguy]

I have to commend you for your effort. I havent seen any other reviewer showing any other usecase than media comsumption, synthetic benchmarks and video encoding and editing. You are perhaps the only youtuber I know who tries out other things like code compile time and ML workloads, which is what is going to run on majority of the high end machines.

[@AZisk]

Glad it was helpful!

[@SimoneFolador]

Thanks about the video man! I loved it and it helped me a lot since I wanted to try some models on my machine. What's your experience on fans on the M3 Max machine? I've read that they are pretty noisy and it becomes pretty hot as well. I still have an Intel machine (last generation) with 64GB ram and 2TB drive but i wanted to buy a new M3 max

[@ChitrakGupta]

That was really good. I learnt something and was fun to run on the new M3 Max

[@gargarism]

I think the very first thing I will try out on my already ordered M3 Max, will be to follow what you did. The whole reason I bought the M3 Max is to work with machine learning. So thanks a lot!

[@AZisk]

Good choice!

[@zt9233]

@@AZiskis m3 max as good as nvidia for this?

[@pec8377]

@@zt9233 no it's not, unless you want to run large model that won't dit into nvdias cards, they Will Always beat M3 GPU. Maybe not when ANE IS activated, but none of thé tools présentes hère supports core ml

[@MikeBtraveling]

If you are looking for a laptop to work with LLMs on you cant really beat the Mac for models larger than 7bP and you want them to run locally@@zt9233

[@camsand6109]

Glad i subscribed. you've been on a roll lately (new subscriber).

[@DivineZeal]

Great video! Thinking about getting the MBP M3 for llm

[@user-kj4ik3qm9d]

Thank you so much for making this video, it was really helpful. Please do more this kind of coding videos and testing on m3 macbook, and push them to the limits, I think you are the best channel for this because you have the knowledge and intention to do these things and it will be win win situation for both of us

[@sujithkumar8261]

Are you using macbook m3 base variant?

[@SergeyZarin]

Thanks great video explaining !

[@AZisk]

Glad it was helpful!

[@joshbarron7406]

I would love to see a token/second benchmark between M2 Max and M3 Max. Trying to decide if should upgrade

[@abhinav9058]

Hey did you upgrade?

[@kman41000]

Awesome video man!

[@AZisk]

Glad you enjoyed it

[@TimHulse]

That's great, thanks!

[@Mrloganphillips1]

I had so much fun with this project. I just got a m3max and wanted a project to work on. After I got llama running I made a bash script to run the command and trigger a second bash script to open a browser window to the ip address after a 5s delay to let the server get up and running first. then I made a shortcuts button to run it. now I have on demand llm with an easy to use on/off button.

[@justisabelll]

Great video, really looking forward to the next few ML related ones. You might have had better results with LM studio though if you disabled mlock after enabling Metal GPU. Also the model output looks nicer if you enable markdown in the settings as well.

[@yinoussaadagolodjo4549]

How to disable mlock ? Can find it !

[@amermoosa]

amazing. just shrinking the whole second grade of engineering college in 17 minutes. incredible 😊

[@eldee8704]

Awesome tutorial! I bought the 14" MacBook Pro M3 Max base model for this to try out.. lol

[@kingmargie1182]

Great job!

[@pbdivyesh]

You're a good lad, thank you!🎉😅

[@theoldknowledge6778]

This LM Studio is Lit 🔥

[@jigyansunanda]

looking forward to your training models video

[@juliana.2120]

ohh i love that you use conda here because it really helps me keep my hard drive clean with all those different AIs :D im an absolute beginner so i'm afraid of installing stuff i cant find later on. some people say its "outdated" and runs in errors too often but i cant really judge that. is that true?

[@vincentnestler1805]

Thanks!

[@AZisk]

🤩 thanks!

[@astrohgamingZero]

Looks good. I use text-generation-webui and the chat/chat-instruct modes or input presets can make or break some models.

[@saitaro]

Thanks for the video, Alex! How does M3 Max compare to M2 Max for ML?

[@user-th8rb5gz3p]

Alex, thanks.

[@AZisk]

You bet!

[@tomdonaldson8140]

Love it! Looking forward to the training video(s). Now I want a Mac Studio M3 Ultra! Oh, no such thing yet? Come on Apple! We’re waiting!!!

[@stanchan]

The performance of the M3 is amazing. Waiting for the refreshed Studio, as the M3 Ultra will be a beast. Hoping it will have the 256GB RAM as predicted.

[@stephenthumb2912]

thanks for testing. it's interesting that even with enough memory, still some slowness on the bigger model quants. my base M2 8gb can run the q4 7b's barely.... prefer ollama using cli which will run at usable tps. it's sort of ok with LM Studio, but generally I need to run 3b's or below with q4 quants. Orca-mini 3b is sort of the default test standard for me on 8gb mac's incl. the mac metal air. can confirm, using the mac metal checkbox, causes runaways. textgen funnily runs fine with mac metal suport as well.

[@MikeBtraveling]

I bought a maxed out M3 max to do this, please run the larger models with ollama, when using LM studio you need to make sure you are using the correct prompt template for the model, i think that was your issue.

[@timelesscoding]

Interesting stuff, I wish I could understand a little more. Thanks

[@AliHussain-jh3iq]

Insightful video Planning to get a MacBook Pro M3 Max for LLM work. Should I go for 1TB or 2TB, 14 or 16-core CPU, and 64GB or 128GB RAM? Thanks for your insight!

[@JunYamog]

Thanks for this content, more dev tilt which is useful for me. I am contemplating on getting MBP after giving my old dev MBP to my niece. Based on your video it seems I would be best to buy as much ram as I can afford and roughly 30B models would need 32Gb shared ram. Possibly better than PC as limited by VRAM? Also I wonder how practical compared to using a cheaper MBP and renting out GPU cloud for the occasional big models? Got more budget constraints.

[@mercadolibreventas]

Kep it up! Good Job! Can you do a video on getting Llama Factory set up on the M3, Thanks!

[@Xilefx7]

Can you test the LLM perfomance in low power mode? I believe Apple needs to optimize how they handle the thermals of the MacBook Pro with the m3 max.

[@BenWann]

I couldn’t agree more - I wanted to really sink my teeth in ML since it’s been a while - and I bought a MBP m3 max after seeing your comparisons. Sorry I couldn’t use an affiliate code - micro center had a killer deal on it :(. I look for your videos to drop now, and look forward to what you come up with next.

[@uninoma]

cool thank you !!!!🤟

[@innocent7048]

Very interesting article. I will try this :-)

[@AZisk]

🤩 thanks so much!

[@salahidin]

Yesss he did it!!!

[@justingarcia500]

Hey could you do a low battery mode test on the m3 max as you did with your m1 max a while back

[@abhinav23045]

That fan noise is like feel the power of AGI.

[@AZisk]

😆

[@juliana.2120]

have you used localAI yet and would you recommend it if so? as far as i understand it uses the same API formatting like GPT so its working with a lot of already existing GPT tools

[@christopherr8441]

If only we could directly access and use the Apple Neural Engine for doing things like this. Imagine the speed and performance gains.

[@davidpsp89]

super interesting and useful, I take this opportunity to ask about Matlab again and its real performance, since Apple's on its page is not real

[@user-wg3rr9jh9h]

Best LLM build video KZbin ❤. I’m buying my 36GB MacBook Pro M3 Max 14 Core cpu with 30 core GPU. Planning on launching a KZbin AI/Ml channel soon 🧐.

[@Stewz66]

If you had the M3 Max, 128GP/4TB, and you wanted to do data analysis and visualization in python, which LLM would you use?

[@paulmiller591]

Great video. Any chance you could revisit LM Studio now? Is it better supporting M3? I am considering swapping out my old Intel MacBook Pro and I do Generative AI development work.

[@AmpiroMax]

Please, can you compare tokens generation speed of any llama-like model on m3, m3 pro, and m3 max?

[@Renzsu]

Hey Alex, can you please do a video on Stable Diffusion on Mac? I'm on the fence on getting one.. the shared memory is tempting, it allows for more than using a discreet GPU. But I wonder how the speed is..

[@user-ob7fd8hv4t]

Is it the 96GB version of the M2 Max, what do you think, I want to deploy my own 13B model locally (train the model with some relatively sensitive data), or even become my 'digital clone', do you think the 38c 96GB M2 Max is a suitable choice?

[@TarunKumar-yf8sj]

Can you please tell how good will these LLMs manipulation with be carried out by M3 Pro Macbook Pro 16 inch.

[@jakubjan44]

good stuff!

[@chillymanny714]

This is a great video, I think if you were to make videos to teach intro/intermediate data analyst how to build LLMs or a series of videos to try different application creation using Macs M chips, that it would be a big hit. I will try to replicate your approach

[@syedanas2083]

I look forward to that

[@muhammadyounis7090]

Hi Alex, thanks for the great content. currently I'm planning to buy new macbook M3 Max for AI work, and I'm hesitating between M3 Max 14/30 with 96GB Ram and 16/40 with 64 GB Ram, they both have similar price tag so I'm not sure to go for the extra 10 GPU cores or extra 32GB of Ram. note that I'm not a pro user (yet) but I want to be able to run locally on new models, train or fine tune my own models with ease, and of course I plan to have the machine with me for 5 years to come.. what should I choose? thank you!

[@anastassogoldschmied]

Can you run LLMs like Llama or Mistral on the Apple Silicon NPU? I think it was possible with stable-diffusion but that is a completely different thing.

[@aalhaimi]

Alex, thanks so much for this.. Quick Question: When running the batched-bench command, I noticed that there are no benchmarks printed under the corresponding table. The table is coming out empty. Everything else seems good.. Any idea why?

[@marabgol]

Thanks Alex! great videos watched 2 so far, do you have videos or plan to make videos how to fine-tune Llama2 models on Metal ?

[@AZisk]

Not yet! but considering digging more into this area on the channel

[@radnaut]

So very awesome 😎

[@matthieuhenocque7824]

Hey Alex, thank you very much for this very instructive video. I've been trying a bunch of local models to help me in my javascript development but couldn't find any that meet my needs. Do you have any recommendation to make? I'm looking for something able to convert ES5 code to ES6, and something able to replace jQuery with Vanilla Javascript. To be honest I have absolutely no clue how difficult it would be for LLM to process my demand, so don't be too harsh on me ^-^ Anyway, thanks a lot for your videos!

[@AZisk]

honestly i haven’t found a local model yet that works even nearly close to what chatgpt can do. but i haven’t done extensive testing with the larger models like 70B+

[@ericadar]

I dont have a local machine with the right specs. What do you recommend for running studio LLM on a cloud instance?

[@randomdude6205]

Awesome video! Any comments on how does it compare with some modern Nvidia GPU? Also, what about training times for small-ish models?

[@tacorevenge87]

Training models locally isn’t efficient even if you do on a pc with the latest nvdia. Why not do it on the cloud ?

[@hksjacky1]

Hi Alex, what is the app that monitor your cpu temperature?

[@redgenAI]

@Alex Should the m3 max with 128gb be able to run a 70b model? And what do you think is the largest model it could fine tune with qlora?

[@emerelle3535]

I bought one expensive m3 max a while ago for getting productive. So far I was not productive and used it for E-Mail and Browsing. At least Safari stopped complaining about memory stuff. (I might haven chosen "slightly too much" compared to my m1 air base model with 8gb) Maybe this video will make me want to finally get productive. :) Edit: As total newbie.. If I do use that LM Studio stuff, just to try things out, do I have to install other thinks like in the video sections before? Or ist it really that ease and just everything kept within that ML Studio Software?

[@hamiltonwmr189]

If you are going to do any intensive task on MacBook then keep it charged at 80% using al dante. Dont let run the models on battery as churning though cycles will damage it's help ,keep it on power adapter and 80% charging. I did some intensive training on my m1 Pro and it went from 100 to 96% battery health in 1 year.

[@CitAllHearItAll]

4% loss in 1 year is normal. I'm at 2+ years on M1 Pro with 86% battery health. You're either trippin or trollin.

[@bern_stock8946]

Can you please do a tf installation video for m3?

[@TheMetalMag]

it's written MBP 2 on your terminal on your MBP3? that's a big job again. You're well into that dev stuff. well done

[@AZisk]

yep. i have to properly name my machines :)

[@ergun_kocak]

3 to 5 times faster than M1 Max 64GM full spec. Thank you very much for the video 👍