Quantization vs Pruning vs Distillation: Optimizing NNs for Inference

[@thomasschmitt9669]

This was one of the best explanation videos I have ever seen! Well structured and right complexity grade to follow without getting a headache. 👌

[@muhannadobeidat]

Excellent video. Well spoken. Nice visualizations.

[@420_gunna]

This felt very nicely taught -- I loved that you pulled back a summary/review at the end of the video - great practice. Please continue, thank you!

[@vineetkumarmishra2989]

wonderfully explained !! Thanks for the video.

[@carlpeterson8279]

Great summary/outline at 17:16 This video covers a lot of relevant topics for neural networks and edge AI.

[@jokmenen_]

Awesome video!

[@heteromodal]

What a great video! Thank you!

[@huiwencheng4585]

Fantastic introduction and explanation !

[@bonob0123]

that was really nicely done. as a non-expert, I feel like I can now have a great general idea of what a quantized model is. thank you

[@ljkeller_yt]

Great format, succinctness, and diagrams. Thank you!

[@RamBabuB-r9s]

your teaches so excellent.. we accepted many more videos from your side to understand for the fundamental NLP

[@kevon217]

^

[@unclecode]

Great content, well done. Please make a video for ONNX, and another one for Flash Attention. Appreciate.

[@kevon217]

Thanks for this!

[@xuantungnguyen9719]

You are a good teacher

[@AmishaHSomaiya]

Great summary, thank you.

[@jeremyuzan1169]

Great video

[@xiaoxiandong7382]

super clear

[@MuhammadAli-dw7mv]

nicely done

[@tosinadekunle646]

Thank you for the video Sir. So please, is quantization just about feature engineering task of data types enforcement of enforcing data types that take only small space? Or it is more than that?

[@EfficientNLP]

I'm not sure if this is what you're asking, but model quantization is not related to feature engineering or enforcing data types; it is methods for making a model more space or compute-efficient after training.

[@tosinadekunle646]

@EfficientNLP But looking at what you have shown in the examples there, it says changing data types to for example int8 for both the input features and the weights. Which i think can be done in a line of code like model.weight.torch.int8. Looking at that, it looks like we must try to ensure that the dataset is stored using a data type that utilizes less memory and to ensure that this is done before model training. What do you think sir?

[@DurgaNagababuMolleti]

Superb

[@hrsight]

nice video

[@yunlu4657]

Excellent video, learnt a lot! However, the definition of zero-point quantization is off. What you're showing in the video is the abs-max quantization instead.

[@EfficientNLP]

The example I showed is zero-point quantization because 0 in the original domain is mapped to 0 in the quantized domain (before transforming to unsigned). In abs-max (not covered in this video), the maximum in the original domain would be mapped to 127, and the minimum would be mapped to -128.

[@ricardokullock2535]

And if one was to quantize a distilled model? Is the outcome any good?

[@EfficientNLP]

Yes, these two techniques are often used together to improve efficiency.

[@julians7785]

I heard multiply by 0 operations are faster to process. Are you sure all operations take the same speed?

[@EfficientNLP]

Generally, for most instructions and most hardware, arithmetic takes a fixed amount of time per operation and does not get faster if the inputs are zeros. However, multiplying by 0 could be faster if the software logic checks for zeros to skip some operations, like in sparsity-aware methods.

[@julians7785]

@@EfficientNLP Thanks for the reply!

[@nothingtoseehere5760]

I am extremely grateful for this detailed explanation, but I am left with a lot of questions. Can I pm you?

[@EfficientNLP]

Sure, happy to discuss over LinkedIn!

[@andrea-mj9ce]

The explanation for distillation remains at the surface, it is not enough to understand it

[@EfficientNLP]

If you have any specific questions I’ll try to answer them!