Deep Dive: Quantizing Large Language Models, part 2

Quantization is an excellent technique to compress Large Language Models (LLM) and accelerate their inference.
Following up on part 1 • Deep Dive: Quantizing ... , we look at and compare more advanced quantization techniques: SmoothQuant, GPTQ, AWQ, HQQ, and the Hugging Face Optimum Intel library based on Intel Neural Compressor and Intel OpenVINO.
⭐️⭐️⭐️ Don't forget to subscribe to be notified of future videos. Follow me on Medium at / julsimon or Substack at julsimon.substack.com. ⭐️⭐️⭐️
00:55 SmoothQuant
07:00 Group-wise Precision Tuning Quantization (GPTQ)
12:35 Activation-aware Weight Quantization (AWQ)
18:10 Half-Quadratic Quantization (HQQ)
23:15 Optimum Intel
25:45 Accelerating Stable Diffusion with Intel OpenVINO

[@cybermanaudiobooks3231]

Great info. Well presented. Learned a lot. Thanks Julien!

[@juliensimonfr]

You're welcome !

[@bashintosh107]

Thank you so much ! Very clear and helpful ! Does the slides are available somewhere ?

[@juliensimonfr]

Sorry, no. You can find the graphs and tables in the research papers.

[@itayatelis2898]

Amazing! Julien do you have any plan on doing a distributed parallel algorithms? (Data par, inter layer, intra layer etc)?

[@juliensimonfr]

Thank you. I did cover some techniques in kzbin.info/www/bejne/sIu4Zn2gd6xknKs. Anything else you'd be interested in?

[@itayatelis2898]

@@juliensimonfr Yes maybe touch on the concept of superposition?