QLoRA is all you need (Fast and lightweight model fine-tuning)

[@Trahloc]

Your idea of having dedicated qlora models as experts is fascinating to me. One of the things i like doing is having AIs emulate a "Council" of various historical figures. Like having Marcus Aurelius, Einstein, and Prince "sit" in a roundtable dicussion of whatever random idea i have. I can only imagine having a specific high quality lora dedicated purely to Marcus's work would improve his emulation.

[@danielpaull309]

Idea. Have multiple qlora models and use either an LLM agent or RL agent to choose which model to use for tasks, then use the different model for inference, or string them together to complete complex tasks

[@Prof1Patel]

Would anyone be interested in making this idea happen? Has anyone tried this?

[@Trahloc]

@@Prof1Patel I haven't heard of one. I haven't even heard of a way to load multiple qlora's and activate them individually for each instance. My coding skills are unfortunately too weak to pull something like that off to do it myself. Best I can do is prompting the emulation.

The MoE made with several QLora adapaters, I have not heard of it before. But the concept of using diffferent QLora adapters in a 'Nesperesso' approach where you provide a "Market Place" of adapters and you customize the generic pre-trained model on the fly depending the context, ... I have implemented. Of course, you cannot switch between adapters for each interaction human-model but at the begin of a session, it is totally doable.

[@Trahloc]

@ Any chance you could try switching the qlora to a different one without reloading the entire model? If the delay for that is relatively short due to keeping all them in ram it'd be worthwhile. Ask a question, present an idea, then let it run overnight locally talking to itself and you can then use something like claude to give a summary the next day? How feasible do you think that idea is?

[@GenAIWithNandakishor]

You need more appreciation than any AI gurus.

[@juhotuho10]

AI "Gurus" vs an actual expert who has done this for years

[@heyyounotyouyou3761]

I think he gets that appreciation too. Just look at his subs. For such deeply tech oriented channel it's difficult to get over a million

[@mshonle]

Appreciation is all you need!

[@GenAIWithNandakishor]

@@heyyounotyouyou3761 but he was one of the great minds in programming community. Underrated I would say

[@connorsheehan4598]

i feel you, the space is filled with low quality content. the karpathy videos great though

[@1olp1]

i feel exactly the same as you concerning how i want a chatbot to be like. Excited to test your wsb model.

[@luvablezombie182]

LoRA has been around for generative image for months. Cool to see we got them for LLMs now.

[@fedrosfieros693]

Having models individualise will definitely allow them to be distinct from other models out in the market and form a sense of personality / humor (and if trained with British data, some sarcasm too). It will allow them to be more "realistic" especially if trained in a way that allows them to conceptualise the information they receive. I think retaining their conversations and then training the models on those conversations will be the way to do that. Thank you for this video, great to watch and precise to its points.

[@antopolskiy]

that's awesome man, thank you for sharing this. if you dig deeper into that, I'd love to see a more step-by-step tutorial, maybe highlighting some aspects or nuances of the process, some theory behind it, etc.

[@isaacjohn9629]

LoRA in SD1.5 and SDXL models is HUGE because its easy to train, easy to use ( WebUI ), easy to swap or stack and easy to find with civitai. QLoRa for LLM needs the same requirements especially the Adapter you were talking about. I Would love a QLoRa trained on Text based adventure games (Correct me if I'm wrong , still New to LLMs)

[@NeoShameMan]

Lora were first invented for llm ironically enough. Not Lora, but the first case of fine tuning was ai dungeon 2.0, to emulate text based adventure, which was the inspiration for current chatbot architecture.

[@mikeyjohnson5888]

@@NeoShameMan Aidungeon 2 used GPT-2

[@NeoShameMan]

@@mikeyjohnson5888 no shit Sherlock, can you make a distinction between methodology and model?

[@mikeyjohnson5888]

@@NeoShameMan Dude, I wasn't arguing. Just adding that was the model used so people would know for posterity's sake.

[@borntodoit8744]

​Nerd fight ! This sh#ts gonna get real...

[@flawedlogic342]

Honestly man’s a genius

[@johnblomberg389]

I have played around with your 13B model and converted it to llama-cpp (quantized 8 bits)... I am just blown away by how natural and lifelike the response are compared to anything else, even the GPT-4 responses seem very robotic and "HR" like in comparison. I can now run inference on my crappy laptop using just CPU and RAM, I don't even have a dedicated GPU on this machine. It is surprisingly quick with an average inference time of about 10-15 seconds (this is with the 13B model, the 7B is very fast at like 3-4 seconds max). The next step would be to increase the dataset with fresh data gathered from the forum and run another QLoRA pass. Also I was thinking about having an active bot posting on the forum, then harvest the responses and how many upvotes the bots get. This data could then theoretically be used for additional RLHF training (although I have no idea on how to do that yet lol). Anyway thanks for this!

[@navinpatle7651]

I am trying to build a multi class classification model for my work, I have minimal data (3000 rows of labelled data), can I utilise the llama 7B for this somehow? Can you share more info on how you converted the 13B to llama-cpp?

[@AIDummy101]

Thank you for sharing this information! Fast and lightweight fine-tuning is a topic that many of us in the AI community are interested in.

[@Endelin]

A Norm MacDonald LoRA would be awesome.

[@akashkarnatak6581]

Yoo, I have been sleeping on QLoRA for a long time. Will try it soon. Nice vid

[@KEKW-lc4xi]

I mostly use GPT for sentiment rewriting, for example, "rewrite the following text better" or "rewrite the following text more eloquently" (if I want to sound super smart), etc. I would love to see how these types of instructions could be trained on open-source LLMs.

[@kindoblue]

Always super interesting content from this channel. Thanks 🙏

[@_rqd2]

Not sure if intended, or some side effects of compression or something, but to me it seems that the video introduces some micro stuttering, at least with 60fps quality options. It's especially visible when you're in motion. I understand a thing or two about cameras, and would suggest checking your camera's shutter speed if it's at a too slow setting (if the camera has the options for changing it, ofcourse). A good rule of thumb for videos is that the shutter speed should be at least double the intended video framerate, so a minimum of 1/120th of a second for 60fps video. I've noticed the fps thing already a while back when watching your videos, and don't know if it even bothers you or anyone else. Anyways, just decided to try and give constructive feedback if it'd help. Don't hate plz 🤓And thanks for the content, good stuff!

[@whoisabishag3433]

DhanOS will improve the SentDex Avatar, thanks

[@prabhavkaula9697]

Thank you for this awesome video!

[@coderaiders-yt]

🤣 - I LOVE that you chose WSB to train this. Legend.

[@bogdankapusta6336]

Thank you. Congrats on keeping that hair line 🎉

[@Wolfwuf]

Awesome video! You should make a video about the GPTQ algorithm and autoGPTQ. Unlike QLoRa however, you need to finetune the model before applying GPTQ since its a one shot. However it provides faster inference. You could finetune a model using QLoRa and then use autoGPTQ to upload it on a consumer GPU

[@ashu-]

Gptq is a quantization technique. what to make video about it ? And it became popular cuz of llama.cpp and then they themselves moved to gguf, which is future proof cuz it retains meta information.

[@Wolfwuf]

@@ashu- you are right, i forgot to mention that i was referring to quantization using QLoRa and bitsandbytes And its not something to do with the video fully, just something within that realm. My apologies 🙏

[@ashu-]

Cool bro

[@JackWelsh-lh8om]

Hey awesome video. I have a few of questions: 1. Why is LoRA (QLoRA) applied only on the q and v matrices of the transformer? 2. What is LoRA applied only on ∆W, and not on the weights themselves? 3. Why can't LoRA be used for pre-training, but only fine-tuning? Thanks :)

[@SpaghettiRealm]

@sentdex you’re all we need in AI ecosystem

[@princemars6746]

Facts

[@PriyanshuTiwari-lh9gm]

content is all you need

[@qwertasd7]

You can instruct a model to have emotions and even emotional goals. I tested this asked it to create a list of emotions and physical needs handle some of these as a time function (hunger sleep etc) others based on the discussion. Then tell it to keep a reward cost system and then it can get quit real. I tried it multiple times and it worked sometimes, better might be to add instead of a point system a simple neural reward system. Easier for time based emotions. Best would be I think to have some long-term memmory ( value keeping system so it doesn't forget values or hallucinate them)

[@coolsai]

Great video Really you're doing amazing work 🎉

[@ander300]

Part 10 of Neural Net from Scratch, about analytical derivatives??? Please bring the series back!

[@jonasls]

I really would love to read/watch more about the MoE Lora project! Where can I follow this?

[@morthim]

gpt 4chan was legendary.

[@johns3195]

Love ur channel. Good content!

[@saw6053]

With llama2-7b-chat it is quite easy to make it have a personality with just adjusting the system prompt. I've had a blast with with something like "You are a cat named Sir Sire, you only know about cat-stuff." Very funny responses including a lot of emoticons etc. Try asking it about (atomic) bombs in example.

[@opusdei1151]

Wow really nice explanation

[@sentdex]

Thanks!

[@calmodovar]

Thanks for the video! It would be great if you can create a video on using QLoRa to fine-tune BERT-derived models on MLM task. These models could also benefit from efficient training techniques developed for generative models.

[@AnthonyBatt]

I enjoyed you video and I too want a less corporate chat model.

[@prestonmccauley43]

This is good stuff it almost connects the last two missing pieces in my brain, I’ve fine tuned some Lora adapters 1. To use them as an llm do I need to merge these all back together to get a gguf? File I can use in day lm studio? 2. If that is not the case is there a sample colab or script that shows how to use the base model+ weightings and is this the preferred technique Thanks!

[@Yarflam]

Thank you so much for this video! I need to training my model now and I'll use QLoRa. Good timing. Did you tried the model "TheBloke/Mistral-7B-OpenOrca-AWQ"? The original model (Mistral 7B Orca) it's a bit better than Llama2, the size is really small (

[@SuperLazyCat]

Lol the response from the AI wasn't wrong though 😂

[@willemmulder9749]

Hi, you mentioned 'skunkworks' ... Is this a private internal project team or a third party involved in similar tech? Thanks

[@kopasz777]

Could the problem be related to the original llama EOS token having an unsupported value of -1? I heard choosing a rare token like 18610 could work.

[@ryanshrott9622]

Next video should be: VLLM and AWS is all you need :)

[@ShubhamShubhra]

You have changed my life bro. Been worried about AI and how it would transform and disrupt our lives since 2018. The only one with any actual clarity on the subject which could resonate with my mind was you.

[@Blooper1980]

Awsome.. Thanks for this. ❤❤❤

Very cool video :) Every time I watch your videos I get curious thought about what is the story behind the USB hub on the microphone arm. :D Also if this trains so fast and with so little data, can this be run locally on a 24GB GPU?

[@sentdex]

Yes you can absolutely QLoRA fine-tune a 7B model on a 24GB GPU (assuming relatively recent for quantization requirements). I think you need ampere, so 3090 or 4090 for consumer GPUs, for example. The USB hub is there to combat the "Yeti whine" issue with this particular microphone. It allows me to add USB power which seems to help make the whine go away.

[@nguyennhi8524]

thank you a lot!

[@alx8439]

Speaking of running on your phone there are some promising projects like Medusa from TogetherAI and DeciLM 6B.

[@hidroman1993]

Would you use LLAMA-2 or LLAMA-2-Chat if you wanted to make an expert on a certain topic? I imagine it's easier to fine tune LLAMA-2 on some documentation, because you don't have to create a "question-answer" format, you simply make LLAMA-2 read the docs

[@mungojelly]

i haven't played that much w/ the non-chat llama 2 specifically but in general the untrained models aren't so much more flexible as they are just utterly aimless, like if you give them a question they might answer w/ more questions that are similar b/c maybe we're making a list of questions who knows what's going on, the basic orientation given to the chat models that there's user messages coming in w/ intentions and they're supposed to do something in particular is pretty useful generally

[@hidroman1993]

@@mungojelly Agreed, also I've looked at kzbin.info/www/bejne/nWebop-lbq6tnJI , they say finetuning a model is not for making it acquire new knowledge, better to use retrieval

[@tejas__]

Do you have a video were we can learn about these llms from scratch? How to got about learning llms to build actually practical application in real World etc.

[@MrRolloTamasi]

Karpathy himself has an excellent series on yt, concluding with a small GPT from scratch.

[@sentdex]

As Mr Rollo suggested, I can't imagine a better source than Karpathy's video ATM: kzbin.info/www/bejne/oXTGaXmjesdkpLs

[@tejas__]

Thank you! @@sentdex

[@elrecreoadan878]

When should one opt to fine tune instead of using a Voiceflow / Botpress ai bot for Q&A? Also what the difference between qlora and gradient is?

[@princemars6746]

19:32 i feel like gpt-4 does that when a person decides to use a persona in the prompts. Especially when using custom instructions.

[@sharangpai4124]

Have you considered or tried running qlora on a mobile device? We were looking for good tech interfacing for this and weren't able to find many good ways to get started.

[@alicenNorwood]

Thanks!

[@sentdex]

Thank you!

[@themax2go]

what we need is "adaptive personality" on top of a LLM... kinda like LoRA, but not pre-trained, and each one always training to develop their own personality... just like humans are

[@NeoShameMan]

Compressor are predictor, in between gzip joint probability compression and quantization, we will arrive at this middle point where we do lossy compression as learning 😂😂

[@drager980]

Dude these qLora models can be reduced down to 9mb even lol it's so good

[@echofloripa]

About models for mobiles and small devices, did you check Tflite? I'm trying to find if I can modify a llama2 model to run on tflite , does anyone know that?

[@bigbena23]

I was wondering... What's the best approach of "arranging the dataset" so it will reflect more than 1 reply per message? Did you simply concatenated all replies to a single one?

[@matthewwilson5560]

Please finish the neural nets from scratch in python series. Only course that is any good at explaining them, shame for it to be unfished. Love your vids!

[@VitaliyHAN]

❤

[@drager980]

I think a quantized 7b model with an archive of 1000s of 100mb models on cloud or etc would be such a good step for getting these llms local

[@borntodoit8744]

Receipe for killer Llamas

[@drramasubramaniam6724]

Yeah a little more character for chatbot does sound good for me

[@goldenfox27]

is possible train a "expert" on documentation for certain topic and still getting a chat output?

[@Martin-po9sz]

yes, it is however mostly done using prompts and vector db. Not fine tuning.

[@imadsaddik]

I would like to see you try Falcon 180B if possible

[@realsemig]

I don't understand how you can only use the adapters that are 200MB for inference. I understand that when you fine-tune a model using QLoRA, you basically freeze most parameters and only update/ retrain a small portion of the full model. However, when running the inference you still need to load and use the full model or am I wrong? Can someone explain please.

[@quentinquarantino8261]

yannik kilcher has a quiet good youtube channel too. And he himself seems very decent to me. I am not sure why he trained a LLM on 4chan.

[@yarikbratashchuk3386]

I have a question: is it possible to tune it based on xml inventory data, so the model could give answers derived from inventory data?

[@sidehat1655]

Just to explain what rank is to anyone interested: *The basics:* _A vector._ In 2d that's a certain amount on the x and a certain amount on the y. So [1, 0] - pointing towards the x with a length of 1. [1, 0, 0] is a 3d vector. [1, 0, 0, 0] is 4d. _A basis matrix._ This contains more than one vector. When you plot on an everyday graph you have the x pointing to the right and the y up. And you plot your vector against that. Well, you can use 2 vectors to describe that. [[1, 0], [0, 1]] one vector pointing to the right and one up. If you changed that to [[2, 0], [0, 2]] in your basis matrix then your vector [1, 0] would suddenly be twice as long. I hope you can imagine how useful that is in graphics engines. *"Get on with it! Tell me what a rank is!"* Imagine your basis matrix is [[0, 1], [0, 1]] - here is the x and y bases are both pointing in the same direction. So no matter what vector you have you can never enter the second dimension. You're stuck on a line. So, the concept of rank tells you it's 1. Now imagine you have a huge matrix of thousands, even millions of dimensions in a neutral net. If your rank is less than your dimensionality then you're inefficiently wasting computational resources and you have an inelegant solution. If you're using 100 dimensions to describe 3d space then you've gone very wrong. You need to boil it down to 3 dimensions but that's what's in the geometry.

[@khaikit1232]

Do you think LoRa or QLoRa would be useful if I wanted to add new vocabulary to a pretrained translation model and finetune it?

[@BrainSlugs83]

Did you mention something about a GPU sniping script? -- Is that linked somewhere? -- I'm not finding it. Maybe somebody can point me in the right direction.

[@JeradBenge]

We need more models that will call people out. 😂

[@legaldesigndo]

I loved your video. I am understanding a bit more thanks to it. Can you share the colab you used in your learning journey? I would like to follow the video with it. I joined Discord channel in hopes you were sharing it there. I want to try training a dataset in another language.

[@radmilraychev5687]

Any idea regarding hardware specs required to fine-tune LLama2-7b using Qlora? I have read that 12 GB VRAM is enough but I have been struggling to make it work for a while.

[@beratcimen1954]

I've trained a few QLora models. Whenever I increase the epochs it just creates the last token until it rans out of context size. I couldn't solve that issue

[@hemantjain2510]

Can you please make a video on context based Machine learning ?

[@someshfengade9623]

Hi can you please give link to the UI code also??

[@DavidJones-cw1ip]

Any reason you didn't provide the script you used to actually train the model?

[@ryanshrott9622]

How to use VLLM with quantized model?

[@murmeldin]

Can you please make a video about gaussian splattering? I've seen it being implemented in polycam and it seems to be in research right now. It is a technique of photogrammetry which seems to have been trained on a large amount of 3D models to make 3D models from a bunch of photos. Greetings from Germany 😊

[@VijethMoudgalya]

21:04 Hilarious [Terraform i see you]

[@ashu-]

😂😂

[@TheAzraf123]

What are your opinions on mojo/modular

[@nathank5140]

Would you be able to do a video on using llama 2 for AI agents? I tried but couldn’t get anything like gpt-4

[@Dave-rd6sp]

Is it possible to train a Llama 7B QLoRA locally on a 4090? Or does it absolutely need server GPUs? I do SDXL training and have never looked into LLM training before.

[@sentdex]

You can comfortably QLoRA a 7B model on your 4090, yes.

[@Dave-rd6sp]

@@sentdex I might have to give it a go. What's the smallest dataset you used that gave interesting or fun results?

[@AC-zv3fx]

Fun AI! Fun AI! Fun AI!

[@SamiKostilainen]

Could the qlora fine tuning be done locally eg with 4090?

[@connorvaughan356]

Does anybody know if QLoRA can be used to fine tune LLMs for regression tasks? I've seen examples for Classification and LoraConfig has a task type specific to classification but I haven't been able to find anything for regression examples. For example, grading student essays on a scale from 0-100.

[@AtHeartEngineer]

Hell ya

[@jit-r5b]

I'm really curious if anyone knows why he's having issues with EOS token...

[@AykutKlc]

Can I use QLORA or LORA to train e.g. LLAMA2 for a new language (in my case Turkish)?

[@mungojelly]

nope! fine tuning will only surface knowledge that's already in the model, and they took out all of the non-english data from the training set for llama,,,,, you might have better luck w/ the falcon models, i think they have some turkish in their dataset

[@vitorfernandes2406]

Free of speech is amazing!

[@renanmonteirobarbosa8129]

You just forgot to mention there are a million of caveats regarding use-case, model-data fitting and so on... Only if it was that simple indeed hahahaha.

[@johnnguyen6224]

Lmao at your model called u out, 🤣 truly based on reddit

[@ExtremeSquared]

Extra confusing because LoRa and LORA refer to something from an adjacent field, but really have nothing to do with LoRA. Someone made some bad choices in naming LoRA. I could see an absurd scenario occurring where someone utilizes LoRA when designing advanced error correction for something over LORA.

[@marcusk7855]

Mine has a filthy mouth. I named her Lilly. She's the best.

[@guyindisguise]

Does anyone know of a good tutorial to add QLoRA to your own (custom) models? And/or some tutorials that implement QLoRA from scratch? (Preferably in PyTorch).

[@user-cw7jy9zr3z]

What the cost of the fine tuning?

[@sentdex]

Depends on how much tinkering you do, model size, data, and settings, but something like $10 to $50ish.

[@UncleDavid]

hari seldon is prolly some model programmed in Kabbala or some shit

[@SDGwynn]

Hey thank you. This is great. oh do you think you could do a walkthrough of local llm on windows using team red? #underserved #24gbgamer

[@GigaFro]

How much money did you end up spending?

[@sentdex]

Tough to say exactly since I used the same server to actually serve some models for inference for a different project and train different models. I think it'd be fair to say I spent a total of 30 hours trying larger/smaller models and variants of ideas just for fun/exploration. @ $2/hr, it'd be $60. THAT said, to run a single QLoRA fine-tune you need about 4 hours. @ $2/hr that's $8. You can keep going too, or add more data...etc, but it's super cheap. Especially compared to the previous best option, which was a full fine tune and that was always $10K+ for models of this size...and data was a much bigger challenge.

[@siddharth-gandhi]

tbh if people get offended by just this much swearing, it kind of implies how childish society is becoming. great video tho! exciting improvements. just hope that the alignment people can stop decapitating these large models.

[@ScottAshmead]

Interesting perspective when I hear people want to see personality from something non-human ...yet here I am and I want to see A.I. as close to being "Vulkan" as you can possibly get (meaning its function should be focused as a utility).... the more human-like A.I. gets the less human interaction humans may need to be..... Interesting note that I recently heard was "your children's first childhood crush will be an A.I."..... let that sink in .... hence I feel it should focus on being a utility without emotion or the ability to use psychological methods in it's deliver of logic and data

[@comosaycomosah]

😂😂this was great

[@braineaterzombie3981]

I was promised ai but got linear algebra.

[@arun279]

What I hate about ChatGPT and Bing is how much handwringing, boiletplate nonsense, caveats it adds. It feels like being lectured to instead of getting an answer to a question. If ChatGPT was a real person irl, I would not go to that person for anything. So it's good to have models that are trained differently or have actually different personalities.

[@mungojelly]

you can also just ask it to communicate differently w/ you ,, here are some requests that work well for me personally w/ chatgpt "use a broad vocabulary and make obscure references" and i just started using "give me a bunch of new ideas about this that i probably wouldn't have already thought of" ,, gpt4 is substantially better both at following communication style requests & also at having something to say, i wouldn't actually expect very much out of 3.5, but actually you can get pretty good stuff from it just by asking for a second or third draft of something w/ specific style notes or directions, if you automate taking it through a few layers of changes gpt-3.5-turbo can do a lot, just uh, one thing at a time, gpt4 can do more like a dozen things at once which is um difficult to make full use of