Sure! I do warn you, it;s on Neurotechnology, which is a bit removed from ML haha It's on Spiral, you may have to log in: spiral.imperial.ac.uk/handle/10044/1/105363 Title: Hardware-efficient data compression in wireless intracortical brain-machine interfaces

@ thanks

Yep, it’s correct that fake quantization isn’t “true” quantization! To have true quantization, one has to do the “conversion” to actual int8. But I tend not to focus on that API too much, since once you have quantization parameters there are many ways to convert the model, e.g. via ONNX, via some custom method that one builds, etc. For example, many on-GPU quantization kernels rey on custom conversion methods where one does bitpacking. So yeah, I tend to leave that part out, but will make future videos on the conversion options (am working on a project to show all of the different conversion options)!

For 1), fortunately ReLUs are stateless, so there aren't any parameters! As a result, it'll load fine. For 2), fusing of layers in eager mode is currently limited to ReLU, but if you use the more modern FX Graph mode quantization (I have a 3-part series on that if you want, and the doxs are great: pytorch.org/docs/stable/fx.html) that will fuse leaky ReLUs / PReLUs as well! If you really want to do it in Eager mode you can do it manually by writing some complicated wrapper, but I wouldn't recommend it (speaking as someone who did it once). A lot of hardware, e.g. Intel via OpenVINO, does support fusing of "advanced" activations and layers, so it's mainly just an Eager mode PyTorch limitatioN!

Really there are a handful of observers available, the main ones being PerChannelMinMax (best for weight tensors, since it gives per-channel resolution). A good one for activations is HistogramObserver, it calculates the MSE between the floating point and fake-quant activations and uses that to assign qparams, but it is slow. There are observers that allow one to fix qparams at given values (e.g. discuss.pytorch.org/t/fixed-scale-and-zero-point-with-fixedqparamsobserver/200306), but you can also do that via the learnable ones. Reading the observer docs is a good place to start! pytorch.org/docs/stable/quantization-support.html

So the main reason I sometimes don't do it in my tutorials is because how one converts to a true quantized model depends a lot on what hardware one is trying to target. For example, PyTorch's original quantization work was all for edge devices, and doesn't run on GPUs. Their new generation of quantization work is mainly on GPUs in the torchao library, and focuses a lot on LLMs. tl;dr: the best way to convert is very hardware specific. I have very recently open sourced a project that helps one figure out how to optimize one's model to any hardware, and we are building a ton of quantization options into it! github.com/saifhaq/alma

So it depends on quite a lot. For some uses cases, fake-quant is only useful for determining good quantization paramaters, and then one has to actually convert the model to a "true" quantized model to run on integer arithmetic. However, in certain cases, fake-quant in itself is the goal. For exmaple, certain hardware, e.g. Intel, often does the quantied operations in fake-quant space. There are other examples too that are a bit of a mix. During LLM inference, we typically use quantization to compress the weight tensors to pack multiple weight elements into as single int32 value, but the actual matrix multiplication happens in floating point. So the kernel dequantizes the compressed weight tensors on the fly, and does the matrix multiplication in floating point space, and quantization is just used to compress weight tensors to reduce the amount of data being loaded from GPU global memory,.

Because it doesn't actually convert the tensor to a different dtype, it stays as fp32. However, it simulates the effect of quantization via the quantization modules attached to weights and activations.

Sorry for the extremely slow reply, I ended up creating a whole open source project that allows one to deploy one's model to basically any hardware, including Tensor RT! github.com/saifhaq/alma Short answer is, torch.compile with the tensorRT backend is a good way to do it!

I haven't dug through the docs for that specifically, but can say that by default FX won't make those kinds of changes to one's model. There is a difference between the weight tensors, but it's not super obvious from the plots because the plots (for weights) work by taking each channel, quantizing it individually, projecting them onto their quantized integer scales, and plotting all of the channels on top of each other. Something like CLE is hard to spot in that circumstance just from overlain weights. Sorry for the late answer!

Thanks 🙂

Sure, I can do that! I’ll set it into the video schedule!

No problem, I’m glad it was useful!

Sorry for the late reply! I’m not familiar with that error message, but it looks like a fake-quantised layer being ported rather than a true quantised layer. The discussion in this thread may be helpful, but if not please reach out! github.com/OscarSavolainenDR/Quantization-Tutorials/issues/12#issuecomment-2123471131

You can clap every now and then to make visible anchors in the track

I appreciate it! Truth is my hardware is a bit old, I was gonna update that and see if that fixes the OBS sync issues. I’m still getting my setup sorted, but ideally I’ll find a solution that fixes the root cause. But if nothing else works, I’ll have to spend time syncing… Thanks for the suggestion!

@@OscarSavolainen neural networks training on an old machine? How is it possible?) Cloud? Seriously, the problem is not just video sync. Also noise from your keybord and mouse. It can be solved by using mic stand without connection your table. I wish you success, your idea seems interesting.

Sure, I can add it in the future videos and onto the Github! Generally, ResNets are measured on top-1 accuracy (e.g. does it correctly classify any image to the correct class). So far I've only been dealing with one image, so top 1 accuracy isn't a great metric. I'll get some more validation data for future videos!

Sure! I’ll make that video a priority, let’s say I’ll have it in 2-3 weeks!

Joo! The name really gives it away haha. I grew up abroad though. Nice one! My friend Yuxin did her PhD at Aalto, it’s a great uni!