Deep Dive: Quantizing Large Language Models, part 1

Quantization is an excellent technique to compress Large Language Models (LLM) and accelerate their inference.
In this video, we discuss model quantization, first introducing what it is, and how to get an intuition of rescaling and the problems it creates. Then we introduce the different types of quantization: dynamic post-training quantization, static post-training quantization, and quantization-aware training. Finally, we start looking at and comparing actual quantization techniques: PyTorch, ZeroQuant, and bitsandbytes.
In part 2 • Deep Dive: Quantizing ... , we look at and compare more advanced quantization techniques: SmoothQuant, GPTQ, AWQ, HQQ, and the Hugging Face Optimum Intel library based on Intel Neural Compressor and Intel OpenVINO.
⭐️⭐️⭐️ Don't forget to subscribe to be notified of future videos. Follow me on Medium at / julsimon or Substack at julsimon.substack.com. ⭐️⭐️⭐️
02:05 What is quantization?
06:50 Rescaling weights and activations
08:17 The mapping function
12:38 Picking the input range
16:15 Getting rid of outliers
19:50 When can we apply quantization?
26:00 Dynamic post-training quantization with PyTorch
28:42 ZeroQuant
34:50 bitsandbytes

[@itayatelis2898]

Love your content! thank you!

[@juliensimonfr]

Glad you enjoy it!

[@bibiworm]

I have been wanting to understand quantization for a very long time. Thank you! Would you mind sharing the slides please? Thank you.

[@matthewrice7590]

Thanks for this, Julien

[@juliensimonfr]

you're welcome :)

[@road2nohand]

Glorious Content :D

[@juliensimonfr]

Glad you like it!

[@Joe-nh9fy]

Great explanation! I have one question... Is it common practice to regularize the LLM cost function like with L2 to reduce the weight "outliers" while training?

[@juliensimonfr]

I don't think there is a strong consensus. It looks like regularization during fine-tuning can help with generalization. There are new ideas too, like noisy embeddings wandb.ai/byyoung3/ml-news/reports/A-New-Method-For-LLM-Regularization--Vmlldzo1ODIyMzIw

[@jacehua7334]

🔥 🔥 🔥

[@juliensimonfr]

:)

[@monishostwal8255]

what is meant by calibration dataset? is it eqivalent to evaluation set?

[@juliensimonfr]

Pretty much, yes. It's used to figure out the "best" hyperparameter values.

[@monishostwal8255]

okay got it thanks

[@caiyu538]

👍

[@juliensimonfr]

😃

[@joaogalego7229]

Watching this at 1.25x speed. High-quality content as usual. Keep it up, Julien 💪