QLoRA-How to Fine-tune an LLM on a Single GPU (w/ Python Code)

Need help with AI? Book a call: calendly.com/shawhintalebi
In this video, I discuss how to fine-tune an LLM using QLoRA (i.e. Quantized Low-rank Adaptation). Example code is provided for training a custom KZbin comment responder using Mistral-7b-Instruct.
👉 Series Playlist: • Large Language Models ...
🎥 Fine-tuning with OpenAI: • 3 Ways to Make a Custo...
📰 Read more: medium.com/towards-data-scien...
💻 Colab: colab.research.google.com/dri...
💻 GitHub: github.com/ShawhinT/KZbin-B...
🤗 Model: huggingface.co/shawhin/shawgp...
🤗 Dataset: huggingface.co/datasets/shawh...
Resources
[1] Fine-tuning LLMs: • Fine-tuning Large Lang...
[2] ZeRO paper: arxiv.org/abs/1910.02054
[3] QLoRA paper: arxiv.org/abs/2305.14314
[4] Phi-1 paper: arxiv.org/abs/2306.11644
[5] LoRA paper: arxiv.org/abs/2106.09685
--
Homepage: shawhintalebi.com/
Socials
/ shawhin
/ shawhintalebi
/ shawhint
/ shawhintalebi
The Data Entrepreneurs
🎥 KZbin: / @thedataentrepreneurs
👉 Discord: / discord
📰 Medium: / the-data
📅 Events: lu.ma/tde
🗞️ Newsletter: the-data-entrepreneurs.ck.pag...
Support ❤️
www.buymeacoffee.com/shawhint
Intro - 0:00
Fine-tuning (recap) - 0:45
LLMs are (computationally) expensive - 1:22
What is Quantization? - 4:49
4 Ingredients of QLoRA - 7:10
Ingredient 1: 4-bit NormalFloat - 7:28
Ingredient 2: Double Quantization - 9:54
Ingredient 3: Paged Optimizer - 13:45
Ingredient 4: LoRA - 15:40
Bringing it all together - 18:24
Example code: Fine-tuning Mistral-7b-Instruct for YT Comments - 20:35
What's Next? - 35:22

[@ShawhinTalebi]

👉 Series Playlist: kzbin.info/aero/PLz-ep5RbHosU2hnz5ejezwaYpdMutMVB0 🎥 Fine-tuning with OpenAI: kzbin.info/www/bejne/aoOkp32qaMuKpas 📰 Read more: medium.com/towards-data-science/qlora-how-to-fine-tune-an-llm-on-a-single-gpu-4e44d6b5be32?sk=4dccc921ab3bd4adc90248293cb13740 💻 Colab: colab.research.google.com/drive/1AErkPgDderPW0dgE230OOjEysd0QV1sR?usp=sharing 💻 GitHub: github.com/ShawhinT/KZbin-Blog/tree/main/LLMs/qlora 🤗 Model: huggingface.co/shawhin/shawgpt-ft 🤗 Dataset: huggingface.co/datasets/shawhin/shawgpt-youtube-comments -- Resources [1] Fine-tuning LLMs: kzbin.info/www/bejne/m3SZeZdnnauppdU [2] ZeRO paper: arxiv.org/abs/1910.02054 [3] QLoRA paper: arxiv.org/abs/2305.14314 [4] Phi-1 paper: arxiv.org/abs/2306.11644 [5] LoRA paper: arxiv.org/abs/2106.09685

[@mouadkrikbou4596]

well done ! well explained, I am a data scientist as well and love your videos, a lot of work behind the scenes to bring the koncepts in such simple yet interactive way!! many thanks Shawhin !!

[@ShawhinTalebi]

@@mouadkrikbou4596 Thanks! This one took longer than usual to put together, so glad you enjoyed it :)

[@MrCancerbero1983]

This is the best explanation that i've ever heard, thanks for all the work!!

[@Ali-me4tv]

So far the best explanation on KZbin about this topic

[@manyagupta6375]

Your explanations are amazing and the content is great. This is the best playlist on LLMs on KZbin.

[@chris_zazzman]

Amazing work Shaw - complex concepts broken down to 'bit-sized bytes' for humans. Appreciate your time & efforts :)

[@BobTheZealot]

Great content, thank you!

[@user-se8ty5nz7d]

Much appreciate your work, Shaw! The most transparent and logical explanation of Qlora Fine-Tuning, you deserve much more. Wish you the best

[@liubovnesterenko956]

Thank you for this amazing video, great explanations, very clear and easy to understand!

[@ai-for-bim]

Amazing video ! You are the best, man ! Thank you so much.

[@RohitJain-ls2ov]

Exactly what I was looking for! Thanks for the video. Keep going!

[@ShawhinTalebi]

Great to hear :)

[@africanbuffalo]

Thank you Shaw for yet another awesome video succinctly explaining complex topics!

[@ShawhinTalebi]

Happy to help!

[@ai4sme]

Amazing explanation!!! Thank you Shaw!

[@ShawhinTalebi]

Happy to help!

Learned a lot. Great video and very accessible. Well Done!

[@ShawhinTalebi]

Great to hear! Glad it was helpful :)

[@wilfredomartel7781]

❤ really amazing work

[@el_artmaga_]

Great video and your slides are very well organized!

[@ShawhinTalebi]

Glad you like them!

[@aldotanca9430]

Loved this, very informative and clear!

[@ShawhinTalebi]

Thanks Aldo!

[@soonheng1577]

wow, you are the genius of explaining super hard math concept into layman understandable terms with good visual representation. Keep it coming.

[@madhurjindal1364]

Man, you are amazing!

[@Wonderfulworldhahaha]

Great content, Thank you.

[@younespiro]

thank u for sharing this knowledge , we need more videos like this

[@ShawhinTalebi]

Happy to help! More to come :)

[@Blooper1980]

Thanks for this!!

[@mr.daniish]

This was a value bomb!

[@medec021]

Awesome, thank you

[@ifycadeau]

Another fire video in the books!

[@ShawhinTalebi]

Thanks! 🔥🔥😂

[@narendraparmar1631]

Thanks Shaw

[@dennou2012]

Great content!!

[@dhirajkumarsahu999]

Thank you so much, I have one doubt please, even if we set fp16 = True, still the optimization would happen in fp32 right, like you showed at 20:22

[@ccapp3389]

Good stuff

[@manuelbradovent3562]

Great video, thanks !

[@manuelbradovent3562]

Additionally, probably also pruning was performed beside quantization in order to get such a low amount of trainable parameters.

[@ShawhinTalebi]

Thanks for the tip! I'll need to dig into that.

[@jjen9595]

nice video bro

[@pawan3133]

Beautifully explained, thanks!!! When you said, for PEFT "we augment the model with additional parameters that are trainable", how do we add these parameters exactly? Do we add a new layer? Also, when we say "%trainable parameters out of total parameters", doesn't that mean that we are updating a certain % of original parameters?

[@ShawhinTalebi]

I explain how LoRA works here: kzbin.info/www/bejne/m3SZeZdnnauppdUsi=_3PK3Kj4Zxs844qg&t=6 Good question. We do not touch any of the original parameters. This just done to give a sense of the relative computational savings of PEFT.

[@felixmuller9062]

Great Video!! How much GPU memory did you need in the end to fine tune mistral 7b?

[@ShawhinTalebi]

Glad you liked it! It runs on Colab with 12.7GB system RAM and 15GB GPU RAM. I don't think it went above 10GB of GPU utilization.

[@CRCaritas]

Thank you for the video! Just a small question. At the end, how would you run inference with your fine-tuned model? Do you save it first to the hub and then load it again? I'm not really sure how to correctly apply the lora adapter to the original model after fine-tuning.

[@ShawhinTalebi]

Yes, that's how I do it here! There's example code for this in the Colab under "Load Fine-tuned Model" colab.research.google.com/drive/1AErkPgDderPW0dgE230OOjEysd0QV1sR?usp=sharing

[@ahmadalhineidi6414]

Great video and explanation! Thanks a lot. For the code, have you tried to use: from transformers import BitsAndBytesConfig nf4_config = BitsAndBytesConfig( load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_use_double_quant=True, bnb_4bit_compute_dtype=torch.bfloat16 ) and then add that as quantization configs when loading the model? This would include the other aspects from the QLoRA paper, no?

[@ShawhinTalebi]

Thanks for sharing! I'll need to try that out. I remember running into issues when trying this on my first pass.

[@naehalmulazim]

Thank you SO much for covering this, Sir! Small question: If I want to fine-tune a model to understand a new coding language whose syntax is similar to C++, any loose ideas or direction I would go about it in?

[@ShawhinTalebi]

There are many ways you can go about this. While I haven't done anything like that, I'd try taking an existing programming model like CodeLlama and doing self-supervised fine-tuning on example code with docstring-like comments.

[@naehalmulazim]

@@ShawhinTalebi Thank you so much! Whether QLora would be used there or should I not use any PEFT fine tuning and go for a full fine-tune would depend on experiments I guess.

[@Etienne_O]

Thank you for sharing this! Have you tried Fine-tune Mixture of Experts like Mixtral 8x7B. Is the process really different? I want to do some testing of my own in the next week. Do you think this requires the same amount of vram as a 7b model or more? I have a macbook m3 pro max with 128gb of shared memory and a mac studio with 196gb of shared memory.

[@ShawhinTalebi]

I haven't played with Mixtral 8x7B yet, so I don't have much insight. Hope to cover this in a future video :)

[@trsd8640]

Thank you for this great video! If you find a way to get this working on Apple silicon machines, we would love to see a video about it!

[@ShawhinTalebi]

Thanks for the suggestion! Once I get something working I'll be sure to share it.

[@Eliot-nr7zq]

Thank you for sharing this fantastic video! Would it be worthwhile to explore a similar approach using unsupervised learning?

[@ShawhinTalebi]

Glad you liked it! When it comes to fine-tuning, the closest thing would be semi-supervised learning. This could make sense if trying to further train a model on a knowledge base (e.g. sklearn documentation). However, empirically it seems fine-tuning tends to be a less effective way to endow a model with specialized knowledge compared to a RAG system.

[@pisthaoct03]

Thanks for sharing. I have a question, instead of quantized model, can i load the base mistral model and follow this process ?

[@ShawhinTalebi]

Yes, given that you have enough memory for the model.

[@pravingaikwad1337]

Is it like the base model is stored in 4bit and as the data (X vector) passes through the layer that layer is first dequantized and then the matrix multiplication is done (X*W)? And the same thing for LoRA as well? and after we get Y (by adding output of lora and base layer) the W and LoRA layers are again quantized back to 4bit? and Y is passed on to next layer? Also, if the LoRA is at the base of the model, does that mean to update the parameters of this LoRA we need to calculate the gradients of loss wrt all the W and LoRA matrices above it?

[@ShawhinTalebi]

That's a great question. Honestly I'm not entirely sure, but what you said makes sense. For inference weights are dequantized layer by layer so that multiplication is possible with FP16 inputs, and no need to dequant LoRa weights since these are already FP16. No need to compute gradients for original parameters because those are frozen i.e. we treat those as constants.

[@Jordano7000]

Hey would this model work if i wanted to input a DNA sequence for example ATCGTGC and the model to repond with the gene name (for example Gene X)?

[@ShawhinTalebi]

I don't know honestly, but it's worth a try. LLMs have a funny way of surprising us.

[@lalmimaj]

Hi, how long did it take you to you to fintune Mistral in this example?

[@ShawhinTalebi]

Took about 10 min to run in Colab

[@nimesh.akalanka]

How can I fine-tune the LLAMA 3 8B model for free on my local hardware, specifically a ThinkStation P620 Tower Workstation with an AMD Ryzen Threadripper PRO 5945WX processor, 128 GB DDR4 RAM, and two NVIDIA RTX A4000 16GB GPUs in SLI? I am new to this and have prepared a dataset for training. Is this feasible?

[@ShawhinTalebi]

That's a lot of firepower! You should be able to do full fine-tuning with that set up. Perhaps you can try using the example code as a jumping off point.

[@pepeballesteros9488]

What's the loss function for this NLP task? I mean, What is the quantitative measure that determines a good response from a bad one?

[@ShawhinTalebi]

I believe cross entropy is used here.

[@pepeballesteros9488]

@@ShawhinTalebi Cheers, I'll look it out. Amazing content Shaw!

[@jeffg4686]

Any idea if GPTQ support is coming to Mac M1 at some point?

[@ShawhinTalebi]

I doubt it. There is an alternative format that works on Mac called GGUF.

[@jeffg4686]

@@ShawhinTalebi- thanks

[@u04vw9]

Have you solved the mac issue? Thanks!

[@HarshvardhanKanthode]

Lemme know as well, I was pretty bummed when I found out bitsandbytes doesn't work on M2

[@ShawhinTalebi]

Not yet. However, that Llama3 is out I have an excuse to spend more time with it. Hope to revisit this in June.

[@yotubecreators47]

I can't save this video, do you know why, can you please enable saving videos to playlist

[@ShawhinTalebi]

That's strange. Are you still having this issue?

[@yotubecreators47]

No I can save it now :), thanks a lot

[@yangyang1412]

what is the minium vram spec for this tutorial

[@ShawhinTalebi]

Runs on Google Colab using 13GB of memory (6.5 CPU RAM + 6.5 VRAM).

[@gk_12344]

Does it work with GGUF models ?

[@ShawhinTalebi]

I didn't try it, but I'm sure there is a way to do that.

[@samyio4256]

When you say "memory" do you mean RAM or VRAM?

[@ShawhinTalebi]

Both! QLoRA specifically uses Nvidia's unified memory feature.

[@FrancescoFiamingo99]

dear Shaw, i m passionated old guy (i m 54 :)) in AI, is amazing how u can explain in simple words concepts , that even an old Mammoth like me can understand , of course , being a total "artisan" in this field , my job is totally different, i m facing problems that will look very simple at your eyes, usually i ask support to chat gpt 4 to learn , understand and correct, but this argument and some of the python libreries are too recent and not yet in the last version of chat gpt 4 , so i need your help, i m not using COLAB becosue i have already similar set on my machine, (16 + 16 like in your example) and i dowload both the model and the data set in my machine , but i m getting this error : ImportError: Found an incompatible version of auto-gptq. Found version 0.3.1, but only version above 0.4.99 are supported i tried to upgrade my version but seems no :ERROR: No matching distribution found for auto-gptq== (any higher then 0.3.1) how can solve the problem?

[@ShawhinTalebi]

It seems like an issue setting up the environment. You can try manually setting the package versions when installing them on your machine based on the Google Colab code.

[@FrancescoFiamingo99]

@@ShawhinTalebi i will try and let u know tks for feedback

[@nikandr8685]

When I tried this, I got this Exception: Cannot copy out of meta tensor; no data! This happens in this step: NotImplementedError Traceback (most recent call last) Cell In[45], line 2 1 # configure trainer ----> 2 trainer = transformers.Trainer( 3 model=model, 4 train_dataset=tokenized_data["train"], 5 eval_dataset=tokenized_data["test"], 6 args=training_args, 7 data_collator=data_collator 8 ) 10 # train model 11 model.config.use_cache = False # silence the warnings. Please re-enable for inference! Do you have any Idea?

[@ShawhinTalebi]

This link might be helpful: github.com/AUTOMATIC1111/stable-diffusion-webui/issues/13087

[@nikandr8685]

@@ShawhinTalebi Thank you. For others with the same problem: This solved it for me: import sys sys.argv.append("--disable-model-loading-ram-optimization")

[@edsonjr6972]

my guess is that the q_proj has 264M parameters, and thats why it's showing only that.

[@ShawhinTalebi]

Wouldn't that make it 264M trainable parameters then?

[@itchainx4375]

@@ShawhinTalebi The training is for a smaller low rank matrix.

[@itchainx4375]

Not for this reason, you can try changing target_modules to see changes in training parameters

[@BenLewisE]

@@ShawhinTalebi I believe that @edsonjr6972 is right and that the trainable parameters is reduced significantly, because you are not _just_ targeting only certain layers, but also you are using LoRA decomposition to smaller low-rank matrices, and so the 264M is the probably the number of all of the parameters in the `q_proj` layers and then the 2M is the ~1% of those parameters that you are actually training due to LoRA

[@sparkledark3713]

It's 264M parameters because it's the only ones which are trainable. Rest ones are frozen.

[@ShawhinTalebi]

Frozen from LoRA or something else?

[@sparkledark3713]

@@ShawhinTalebi Like the main model parameters are frozen except LoRAs parameters. Maybe that's why

[@sapandeepsandhu4410]

Enlightening journey through the intricacies of Large Language Model (LLM) optimization! 🌌🖥 Your adept presentation not only demystifies the process but also serves as a beacon of inspiration for both burgeoning and seasoned developers navigating the vast seas of AI technology. The elegance with which you delineated the nuances of QLoRA and its transformative approach to fine-tuning LLMs on a singular GPU setup is nothing short of revelatory. 📘✨ It's a masterclass in making advanced AI technologies accessible and practical for a wider audience, empowering individuals to harness the full potential of LLMs without the necessity for extensive computational resources.

[@xi8t-gk1oi]

fp16=true causes training to fail with error "No inf checks were recorded for this optimizer.". Set fp16=false and training successfully completed but loss and eval loss are the same for every epoch.

[@xi8t-gk1oi]

I am trying to fine tune on my own dataset with 20000 messages in format: msg_id, sender_id, content, reply_to, interval (between this and previous message) to generate similar messages with similar format.

[@ShawhinTalebi]

Are you running the provided script in Colab?

[@xi8t-gk1oi]

@@ShawhinTalebi no, on my own machine

[@MrCancerbero1983]

Same here, but training doesn't take effect, i got the same answer after training

[@MrCancerbero1983]

i changed the torch version to match colab via pip install torch==2.1.0, and it work

[@PranavBaviskar]

Getting Key Error: Mistral

[@ShawhinTalebi]

I'm not able to replicate that error. Are you running the example in Colab?

[@Ev4Nou4]

im so fucking lost

[@ShawhinTalebi]

This video goes pretty deep into the technical details. Watching some of the previous video in the series might help give more context. I also do office hours if you have any specific questions: calendly.com/shawhintalebi/office-hours

[@jjen9595]

i get this error OSError: TheBloke/dolphin-2.6-mistral-7B-dpo-laser-GGUF does not appear to have a file named pytorch_model.bin, tf_model.h5, model.ckpt or flax_model.msgpack. ¿que hago? :c

[@ShawhinTalebi]

Not sure, I haven't come across that one before

[@jjen9595]

@@ShawhinTalebi I solved it, you must put the correct model in the colab that is similar to the one you have, I still don't know how to make a meta for hugging face :c