All You Need To Know About Running LLMs Locally

[@bycloudAI]

stay up-to-date on the latest AI research with my newsletter! → mail.bycloud.ai/ Minor correction: GGUF is not the predecessor to GGML, GGUF is the successor to GGML. (thanks to danielmadstv)

[@Sebastian-oz1lj]

please make step by step guide how to install locally and private for example Mistral-7B. im trying to do this with multple guides and all time im stuck at something

[@danielmadstv]

Thanks for the video! Minor correction: GGUF is not the predecessor to GGML, GGUF is the successor to GGML.

[@ambinintsoahasina]

The amount of infos you give both in the videos and the descriptions is insane dude! Keep up the good work!

[@noobicorn_gamer]

I hoooonestly don't know how to feel about the thumbnails looking too similar to you-know-who that got me accidentally clicking this video but meh... One's gotta do what one's gotta do I guess.

[@EdissonReinozo]

Same

[@Dedjkeorrn42]

I don't know who, who?

[@nathanfrandon2798]

@@Dedjkeorrn42 Fireship

[@NIkolla13]

Bycloud removed the frame and the grid background on his thumbnails, I think those work great as his signature style. I hope he keeps them

[@seanrodrigues8184]

Let's just hope he doesn't get _burned~_

[@flexoo7]

Poor Faraday nearly always gets overlooked when people talk about local LLMs, but it is without a doubt the most easy to use "install and run" solution. Unlike nearly all other options it's near-impossible to mess something up and default settings out of the box are not sub-par.

[@hablalabiblia]

How much is Faraday?

[@flexoo7]

@@hablalabibliaLike all the best things in life - it's free.

[@joure.v]

@@hablalabiblia It's free and very easy to use! It's really meant just for chatting, it's basically a Silly Tavern kind of app, just not with that many options but it has its own back end with a focus on GGML models. If you're looking to just run models through character cards I'd say, give it a go!

[@Elegant-Capybara]

Faraday has outdated models and whenever you download models, you have to fumble with model cards and directory structures, plus it's not as fast as other options. LM Studio is better than Faraday.

[@jbol2454]

@@Elegant-Capybara LM Studio is closed source.. no thanks..

[@RetroPolly]

A thousand thanks! Finding a good LLM model was a complete nightmare for me + it is difficult to figure out which formats is outdated and which - new hot stuff.

[@Leo_Aqua]

You can also use ollama. It even runs on a raspberry pi 5 (although slow)

[@siddhubhai2508]

Yeah you're right OLLAMA can be run on the raspberry pi 5 even, but don't forget that ollama is made for using local llms, and if you try to run local llms like llama 3 or deepseek, ready for FBI at your home catching you for building a unknown b*mb. Important life lesson - FIRST TRY THEN CRY. GOOD LUCK! 💣

[@NickH-o5l]

I got Gemma 2b running on my end I got faster token per second with this really small model from alibaba (yes, it’s biased) with 0.5b parameters, but if you ask it right maybe there’s some use case But it’s kinda dumb

[@siddhubhai2508]

@@NickH-o5l Low parameters = Low accuracy 👍

[@아잉뀨잉뀨잉-u5q]

I have been struggling on this issue for few months, and seems like this video already had the answer more than half an year ago. Thank you for your awesome vid!! Really love your work!

[@Veptis]

Now we just need a cheap inference card with 128GB memory to run 70B models locally... Maybe we can hope for Qualcomm

[@cbuchner1]

I’d love to see AI inference accelerator cards with dual or quad channel DIMM slots.

[@Veptis]

@@cbuchner1 Qualcomm AI 100 Ultra is using LPDDR5

[@nyxilos9167]

groq is using something of the sort, an LPU. although only usable through an api. no consumer cards yet that i know of, but it shows the trend towards it

[@Veptis]

@@nyxilos9167 you can buy a single groq card right now. it costs 21k and has 230MB on board. So to run 70B models at fp16 you need like 572 cards.... which is several racks. 14+ million to buy and 30kW to power. It will run the model at 400 tok/s easily. You can buy a ready made 8x H100 box for maybe 350k and run that with like 8kW and it might be slower than the groq card. none of that are consumer solutions. The one I am hoping for is Qualcomm AI 100 Ultra. Which comes with 128GB LPDDR5 and 150W. They say it's for edge inference, but it would be perfect for workstation.

[@Vifnis]

idk Qualcomm SoCs are for phones mostly... maybe iPhone 30 will have it XD

[@CarinaAdele5CA]

The market trend can turn around very quickly. In fact, the indexes often switch from a bear market to a bull market when the news is at its worst and the mood of investors is at its lowest point. I read an article of people that grossed profits up to $150k during this crash, what are the best stocks to buy now or put on a watchlist?

[@AmaliaGiselae8g]

In particular, amid inflation, investors should exercise caution when it comes to their exposure and new purchases. It is only feasible to get such high yields during a recession with the guidance of a qualified specialist or reliable counsel.

[@AlbrechtChristoph016]

True, initially I wasn't quite impressed with my gains, opposed to my previous performances, I was doing so badly, figured I needed to diverssify into better assets, I touched base with a portfolio-advisor and that same year, I pulled a net gain of 550k...that's like 7times more than I average on my own.

[@BerchtwaldElias1EH]

This aligns perfectly with my desire to organize my finances prior to retirement. Could you provide me with access to your advisor?

[@AlbrechtChristoph016]

NICOLE ANASTASIA PLUMLEE’ is the licensed fiduciary I use. Just research the name. You’d find necessary details to work with a correspondence to set up an appointment.

[@BerchtwaldElias1EH]

She appears to be well-educated and well-read. I ran an online search on her name and came across her website; thank you for sharing.

[@Paulo-ut1li]

Boy, Chat with RTX is my personnel oracle for now on. Its RAG really indexes local documents without that whole hallucination from previous tools.

[@chevalier5691]

I don't get these complaints about the thumbnails. Are you guys new to youtube? We have been through the era of fake or nsfw thumbnails and yet you're still complaining about similar style? If you're not willing to check the uploader channel name or profile, then enjoy getting scammed by phishing links online.

[@NeostormXLMAX]

but anyways this video was very helpful because no one made it very clear on what are the best front end interfaces to install, I kept trying to make one myself to no avail and give up after a while after testing stuff in the command prompt

[@bossdaily5575]

Nice video! Can you do a video about fine tuning a model?

[@papakamirneron2514]

Immensely helpful video. I hope the future has tonnes of user controlled locally ran llms for us in store!

[@remboldt03]

Stup osing Fireship thumbnails😭

[@idk-dk7bq]

Y

[@aouyiu]

Stop neglecting proofreading comments 😭

[@remboldt03]

@@aouyiu I apologize. I normally proofread all my comments, but I suspect that I was drunk while writing this one. As I don’t like editing comments afterwards, I didn’t change the spelling mistakes.

[@HPTRUE]

Never heard of fireship....

[@juanantonionieblafigueroa377]

Your videos are way more fun than my algebra homework

[@robertmazurowski5974]

I was pretty sure this was a fireship video, but the video is great and informative. Exacly what I was looking for.

[@magfal]

The one thing I hope to see soon is offloading different layers to different GPUs I have a 4090 mobile in my laptop and an RX6800 in my eGPU. I do have 96GB of system memory in addition to these two 16GB cards so I can do some fun stuff already.

[@H1kari_1]

I love your adhd-friendly edits cloudy.

[@the_gobbo]

I can finally start my side project to take over the world, thanks!

[@johnsarvosky533]

Thanks, this is great. Please make a comprehensive video on Fine-tuning locally 101..Cheers

[@lunadelinte]

that was awesome, thanks for the concise information bycloud! 🔥

[@jawbone1218]

Curious headcount? 🙋How many of us watching these type videos are not developers?

[@proflead]

A video about fine tuning a model would be nice!

[@a.........1._.2..__..._.....__]

Ive been hamfisting my way through llms for over year. Just ramming squares into circles till it worked since informations so sporadic. 100% checking out your other videos. Learned more in 5 min then 4 hours reading github docs

[@trolik9113]

Absolutely fantastic and informative video. Well done! I will say I feel like the information certainly speaks to the grip that OpenAI has, especially from a development standpoint, despite the whole video being about open-source models. The procedures, time, research, and money required for any rando or small (even mid size) business owners to integrate open-source and local AI without any practical knowledge about it is near impossible. OpenAI wraps up RAG, "fine-tuning", and memory nice and neat into Assistants which can be easily called via the API. It would be amazing to have a completely standardized system that allows for the same type of application, but geared towards the variety of open-source models out there. Some platforms like NatDev let you compare multiple models based on the same input. Being able to see how RAG and fine tuning affects different models, both open-source and non, from the same platform would be unreal.

[@pedrogorilla483]

Where ollama?

[@sZenji]

agree, with the new windows installer its so easy for everyone to get local models

[@4.0.4]

For a while it was only Mac-based, so it saw limited use with most AI folks who have Nvidia cards. If you're stuck on a Mac I hear it's really the better one for that.

[@zikwin]

wow now on support windows too ?@@sZenji

[@babbagebrassworks4278]

I use it on my Raspberry Pi5 to run LMM's, which is seriously cool, er hot when working.

[@lintalyor6535]

I like this simple explanation with the video editing thanks!

[@WINTERMUTE_AI]

LM STUDIO and TRINITY 1.2 is my favorite non-GPT entities!

[@vladislava5237]

Very nice, tons of useful info Thank you!

[@plagiats]

Ollama + openwebui is the way to go. Same ui as ChatGPT, plenty of convenient functions. It's a no brainer.

[@NoidoDev]

The important part for me is accessing it from CLI or Python. Ideally, doing the whole configuration in there. Because I need it to be automatized (no NodeJS of course).

[@Kevin.Kawchak]

Thank you for the discussion.

[@WINTERMUTE_AI]

I keep canceling my GPT4 subscription and then renewing it... 'Just when I thought I was out, they pull me back in.' GPT4 reminded me of that phrase from The Godfather. :)

[@DanielHayes-p2u]

The thumbnail style is just like Fireship

[@Anthonyg5005]

EXL2 does support AMD GPUs. Turbo bought a couple just to make sure it runs with rocm

[@hyposlasher]

2:17 Bro lives in the future where M4 is already released

[@artursvancans9702]

You pay 20$ for convenience. Spending 1 day to set up the flow, Waiting 2 minutes every time for your model to load when you have a quick question, your GPU + CPU setting your room on fire cuz of how hot they're running... Unless you need some really specific usecase that cloud models censor, then it's just easier to pay those 20$ for instant access

[@thatguyalex2835]

Patience is a virtue. I got Mistral 7B running on an 2018 laptop, and it takes two minutes to respond, but it works well. Why have 8 GB of RAM when I don't use all 8 GB. The AI uses all my RAM. :) But, for people who have to use AI for a job, $20 is cheap, and workplaces cover the cost. For AI at home, a fast enough computer could work.

[@fennecthechoosenone5189]

Koboldcpp crying in the corner

[@rotors_taker_0h]

Basically to understand this video one should already know everything mentioned in this video by heart.

[@Trahloc]

Eh, it provides terms to hunt for and sometimes that's all someone needs, a starting point. The video is short and covers a lot of ground.

[@MonkeeGeenyuss]

Dude wants a 16 part lecture to explain it all😂

[@rotors_taker_0h]

@@MonkeeGeenyuss I mean, I can only follow because I know it all and cannot imagine someone unfamiliar to understand anything from this firehose, lol.

[@fra4897]

what about ollama as a backend, what is your take on that? Thank you so much for the video, sending love from switzerland

[@zyxwvutsrqponmlkh]

I just really really like how many serious people have to say ooobabooga. It's like, almost as good of a joke on science as when that guy named the seventh planet.

[@valeriapadilla5860]

Hope this works better than the time I tried to download more RAM

[@shoddits2156]

does the a Giveaway has country restriction?? I mean maybe you can't send it overseas due to shipping cost or something else.

[@krzysztofmaliszewski2589]

That's a great question.

[@aketo8082]

Thank you. Very interessting. Is it possible in LM Studio to work with own files? Or create own LLM or extend LLM for own cases?

[@exaltedjoseph7963]

Your thumbnail reminds me of fireship

[@itisallaboutspeed]

as a car content creator i approve this video

[@mzafarr]

Please make a video about making our locally running LLMs available for others to use maybe like our own API which people can use or a webUI interface to use our local LLM.

[@tutacat]

You don't need finetuning, just do more prompts

[@dipereira0123]

To be fair, at 8:53, the 10 bucks you will be "saving" from running you LLM locally instead of paying github copilot will probably become more expensive in your energy bill... (your GPU will be working at max capacity) and lets not talk about the time it will take to set it all up unfortunatelly... the AI rev is something that will be in the hands of big corps

[@abdelkaioumbouaicha]

📝 Summary of Key Points: 📌 The video discusses the landscape of AI services in 2024, highlighting the abundance of hiring freezes and the prevalence of subscription-based AI services. 🧐 Various user interfaces for running AI chatbots and language models locally are explored, including UABA, Silly Tarvin, LM Studio, and Axel AO. 🚀 The importance of choosing the right model format, understanding context length, and utilizing CPU offloading for running local language models efficiently is emphasized. 💡 Additional Insights and Observations: 💬 "Garbage in, garbage out" is a crucial principle highlighted when fine-tuning AI models, emphasizing the significance of quality training data. 📊 Different model formats like GGF, AWQ, and EXL 2 are explained, showcasing how they optimize model size and performance. 📣 Concluding Remarks: The video provides a comprehensive guide on running AI chatbots and language models locally, emphasizing the importance of model selection, context length, and fine-tuning techniques. Understanding these key aspects can help individuals navigate the AI landscape effectively and optimize performance while saving costs. Generated using TalkBud

[@Afro__Joe]

Lol nobody reads anymore by these comments. Ooh shiny picture, click! Thanks for the info, I was looking for a video like this yesterday.

[@seasong7655]

I'm glad I avoided the 20 dollar subscription by buying a $500 GPU and $100 ram

[@rougeseventeen]

thanks, this videos is very funny and helpful!

[@fmachine86]

You're not fireship, cut it out.

[@fmachine86]

@@MusicalMessagges The he should make his own videos and not try to copy someone else.

[@moresignal]

Yep, when you rip off the style of Fireship with your title card you're tricking our brains into thinking we're going to see his content. Develop your own style.

[@MusicalMessagges]

@@fmachine86 In my defense, I realized it was copying fireship after I typed the message and didn't realise I sent it

[@fmachine86]

@@MusicalMessagges Saul Goodman.

[@bungamawar-i8m]

Yeah he's better

[@siddhubhai2508]

Oh Fireship's second hidden channel! 😂😂

[@u13e12]

Just to clarify then. For inference speed is more important GDDR6 will be GDDR5, but for fine tuning more more having 2x the amount of GDDR5 will be the GDDR6?

[@bigglyguy8429]

How did you miss Faraday? Very easy to use and runs faster than LM Studio

[@rumali_roti7406]

You added models in the description but specify their usage. Can you add more details, please?

[@alfamari7675]

So acording to the description llama 3 killed deepseek coder, wizard, and mistral? I just started getting into this stuff recently and those were some of the top performing models I had heard about (though they existed before llama 3).

[@LinkEX]

1:32 I typed "i am new to github" into my search bar, and sure enough, the autocompletion suggested the thread title. Came for the replies, which were more tame and not as many than I had expected. I initially thought this was an older image meme and you merely reused the screenshot. But since the original post was in fact posted 5 months ago (like this video), and the screenshot was shot 15 minutes after the post, I conclude you probably frequent r/github.

[@NeostormXLMAX]

I spent so much time trying to get something like this set up, but ended up back to gpt, most of these models are also censored just like gpt, and unlike gpt they are much slower AND on top of that they canot use plugins or special api's that let you access the internet or generate images etc. its sad but currently gpt has no peer

So, AI is the new computer, everyone will have one? Seems good to me. I wonder how the job market will be, hardware will be on top for sure and Open AI will still being a giant. But the thing is how other industries will be affected.

[@alan_yong]

🎯 Key Takeaways for quick navigation: 00:28 *🤖 Running AI chatbots and LM models locally provides flexibility and avoids subscription costs.* 00:43 *📊 Choosing the right user interface (UI) for local AI model usage is crucial, depending on individual needs.* 02:05 *🖥️ UABA is a versatile UI choice for running AI models locally, supported across various operating systems and hardware.* 02:33 *💡 Installing UABA enables access to free and open-source models on Hugging Face, simplifying the model selection process.* 05:18 *🤔 Context length is crucial for AI models' effectiveness, affecting their ability to process prompts accurately.* 06:12 *⚙️ CPU offloading allows running large models even with limited VRAM, leveraging CPU and system RAM resources.* 06:52 *🚀 Hardware acceleration frameworks like VM inference engine and TensorRTLM enhance model inference speed significantly.* 07:36 *🎓 Fine-tuning models with tools like Kora enables customization for specific tasks, enhancing AI capabilities.* 08:47 *💰 Running local LM models offers cost-saving benefits and customization options, making it an attractive option in the AI landscape.* Made with HARPA AI

[@Paulo-ut1li]

Please make a video on how to fine tune a model using local documents.

[@4.0.4]

Dunno why my comment isn't going through, but try Kobold! Better for GGUF. Current fav is "Crunchy Onion" Q4_K_M GGUF. Give it a taste! 10t/s on a 3090 and pretty smart.

[@violet-trash]

Kind of sucks that the GPU brand that works best with AI is the one that skimps on VRAM. 💀

[@dungeon4971]

what about ollama

[@kernsanders3973]

In regards to context, would LLM Lora's help with that? Lets say im busy with story writer LLM and the fantasy world I'm working with would be as big as something like Middle Earth from LOTRs. Would a Lora help with that? Like if I train a Lora on all our past chat history about the story etc. Also more text regarding the lore of places and history of characters and family trees. So taking that into consideration, would that assist in keeping the context low? So I don't need to keep a detailed summerized chat history etc. What would the requirement be for training such a Lora and what would the minimum text dataset require for a coherent training?

[@squfucs]

i run LM Studio and i think its great, good video my dude

[@Cergorach]

I was highly disappointed, Shōji Kawamori and Kazutaka Miyatake are not on the panel about Transformers... ;)

[@bibr2393]

>this model list

[@cristianionascu]

I guess my machine is not good enough, 2019 intel imac, because running any model locally is usually lagging way behind ChatGPT 3, Gemini, Perplexity, etc.

[@GraveUypo]

running LMs on linux and windows, for some unknown (to me) reason, linux is over 5 times as fast as windows at prompt evaluation. it's not even close.

[@samuelpeery]

Total newbie with running an LLM locally. What is the best llm for summarizing books and being able to ask questions about the books?

[@hardik4942]

What's the best for investigation and data analysis?

[@Zonca2]

Wish you made more down-to-earth guide on how best to chat up waifus in Sillytavern, the community is super small for what you can achieve with minimal knowledge, running something like Noromaid on google collab, for completly free and uncensored roleplay, it needs to get more well known, plus I dont really know my way around the different settings and models, having a hard time to get the waifus to put in more dialogue over descriptions for example.

[@Quell__]

jesus I truly hate how intertwined the ai community is with the anime community

[@kernsanders3973]

Mistral buyout from MS was a huge blow to the open source AI community, the future is looking very corporate controlled...

[@AleNovelasLigeras]

We're gonna need a bigger GPU

[@Saeed_al-moumen]

my brain hurts ( i only reached 4:08 I just watched the video to see if there anything I need to know about sillytavren since that what I searched but i don't thinks there any more )

[@gregNFL]

Stanford open source LLama model is free. 🎉

[@joseph-ianex]

with local models are you able to make much longer responses given that you have enough ram and vram?

[@harrymahon4]

We love LM Studio 😫

[@kingki1953]

I thought it was a video from Fireship 😂

[@RedOneM]

What 3 models do you recommend with 24 GB VRAM? Preferably 21-22GB / 24GB in practical usage.

[@nyxilos9167]

huggingface lists models with their respective memory requirements. any 7b model will likely work very well and be under 21gb. you could also go with a bigger model but at a lower quantization. mistral models are among the most popular, open source, and very competitive.

[@Necessarius]

Fireship thumbnail is working for me

[@keffbarn]

OOOGABOOOOGAAAAH 💪😎🍺

[@zzador]

Damn...I got my 2015 Laptop ready with 4GB of RAM and BAM: Every model needs at least 8GB RAM. What a bummer.

[@christopheralvarez1090]

Where do I upload the photo once GTC comes around ?

[@Frab1985]

The best RP model atm is Kunoichi v2

[@AshishKumar-kv4hr]

Are you the same as fireship?

[@swaggitypigfig8413]

Different human being

[@I_SEE_RED]

it’s fireship experimenting with 100% channel automation

[@XxXVideoVeiwerXxX]

silly tarven....he used AI voice for this

[@NatalyOrtíz-z3m]

Good to know.

[@felipetesta]

Anyway I can set a local AI that can access PDF files from my university folder and help me summarize and introduce the themes I have to study using the PDFs as primary source of content?

[@tja9212]

timecode 1:18 is a very questionable use of footage

[@Evagoesbrrr]

Next step - chub chars. Next step - writing own bots. Next step - quant own models. Next step - uncensor models with ability to update context. Next step - mooooom, i need money for new nasa servers! 256k context is too small! My waifu getting stupid!

[@dustindustir521]

Step 4 is Clear, but How can I unlock step 3? I only see questionmarks. Do I have to do step 1 and 2 to unlock what I have to do at step 3, Or do I just need to gain more XP for the unlock. Maybe I just have to do step 4 twice to make up for the missing third step...

[@knoopx]

lm studio/ollama are probably the simplest ways to get started, not sure why you picked these ones

[@FlafyDev]

this isn't fireship.. where am I?

[@myname-mz3lo]

same . the thumbnail got me and then i realised this guy took fireship's entire style