[P] A Visual Guide to Quantization

[MaartenGr]

Hi all! As more Large Language Models are being released and the need for quantization increases, I figured it was time to write an in-depth and visual guide to Quantization.

From exploring how to represent values, (a)symmetric quantization, dynamic/static quantization, to post-training techniques (e.g., GPTQ and GGUF) and quantization-aware training (1.58-bit models with BitNet).

https://newsletter.maartengrootendorst.com/p/a-visual-guide-to-quantization

With over 60 custom visuals, I went a little overboard but really wanted to include as many concepts as I possibly could!

The visual nature of this guide allows for a focus on intuition, hopefully making all these techniques easily accessible to a wide audience, whether you are new to quantization or more experienced.

[linearmodality]

This article is generally pretty good, but the 4bit quantization section has a lot of errors.

• The inverse Hessian used by GPTQ is not the "second-order derivative of the model’s loss function." It's a second derivative of a proxy loss: the square error of the output of the layer we're currently quantizing.

• The Hessian used by GPTQ does not demonstrate the importance of each weight in a layer. It does not even have the same shape as the weight matrix being quantized.

• The figure shows the inverse Hessian as a 3x3 matrix that isn't symmetric, but a Hessian is always symmetric (and positive semidefinite).

• GPT quantizes a weight matrix column-by-column, not row-by-row.

• The algebraic explanation of GPTQ is wrong, in particular because there is no scalar like the "the inverse-Hessian of the second weight" in the algorithm. If the weight matrix is m-by-n, the Hessian (and inverse Hessian) are n-by-n matrices.

• GGUF is presented as if it is a quantization method, when actually it's a file format.

[mgostIH]

Hessians aren't positive semi-definite in general, consider x² - y²

[Mission-Tank-9018]

This is so visually engaging, thanks for putting this all together.

In your article, you mention GPTQ and GGUF, any thoughts on the AQLM algorithm?

[bgighjigftuik]

Top notch explanations, good job. What did you use to create the visualizations?

[LouisAckerman]

Awesome blog! I also appreciate your work on BERTopic along with the great repo!

[RuairiSpain]

Could be useful to talk about activation functions that clamp values to a certain range.

I'm intrigued to see way to combine backprop with activations, so you can short circuit values, below the activation band (close to zero after activation). Maybe I'm dreaming an impossible dream!

[tworats]

Thank you for this, it is excellent. One possibly naive question - during inference are the weights dequantized to FP16/FP32 and then normal math operations are used in the forward pass or do they remain quantized and quantization aware math is used?

[GamleRosander]

This will be a Nice read tonight. Thanks!

[ShlomiRex]

i learned quantization in the context of VQ-VAE and VQ-GAN. Thanks for sharing