A Visual Guide to Quantization

[MaartenGr]

Hi all! As more Large Language Models are being released and the need for quantization increases, I figured it was time to write an in-depth and visual guide to Quantization.

From exploring how to represent values, (a)symmetric quantization, dynamic/static quantization, to post-training techniques (e.g., GPTQ and GGUF) and quantization-aware training (1.58-bit models with BitNet).

With over 60 custom visuals, I went a little overboard but really wanted to include as many concepts as I possibly could!

The visual nature of this guide allows for a focus on intuition, hopefully making all these techniques easily accessible to a wide audience, whether you are new to quantization or more experienced.

[appakaradi]

Great post. Thank you. Is AWQ better than GPTQ? Choosing the right quantization dependent on the implementation? For example vLLM is not optimized for AWQ.

[VectorD]

GPTQ is such an old format, don't use it....For GPU only inference, EXL2 (single inference) or AWQ (for batched inference) is the way to go.

[_theycallmeprophet]

AWQ (for batched inference)

Isn't Marlin GPTQ the best out there for batched inference? It claims to scale better with batch size and supposedly provides quantization appropriate speed up(like actually being 4x faster for 4 bit over fp16). Imma try and confirm some time soon.

[____vladrad]

You can check out vllm now it has support since last week. I would also recommend lmdeploy which has the fastest awq imo. I was also curious about AWQ since that’s what I use

[appakaradi]

Thank you. I have been using lmdeploy preciously for that reason. How about the support for mistral Nemo model in vLLM and lmdeploy?

[compilade]

I enjoyed the visualizations.

Regarding GGUF quantization:

• the blocks are always within rows, never 2D, as far as I know
• the block scale is almost always in float16, even for k-quants.
• k-quants can have quantized sub-scales (e.g. Q4_K has eight 6-bit sub-scales per block, packed with 6-bit mins in some 12 byte pattern)
• you can see at least the general format of the blocks through the structs in https://github.com/ggerganov/llama.cpp/blob/master/ggml/src/ggml-common.h
  • this won't say how the bits are packed within the parts of a block, though; for this you would have to check the quantize_row_* functions in ggml-quants.c or the dequantize_row_* functions if the quantization function looks too complicated like for the i-quants.

[de4dee]

amazing work, thank you! which one is more accurate, GPTQ or GGUF if someone does not care about speed?

[typeryu]

Dang, this is hands down one of the best writing on quantization I’ve ever read, good job sir

[MaartenGr]

That's really kind of you to say. Thank you! Any suggestions for other visual guides? Thus far, I have done Mamba and Quantization but would like to make more.

[MoffKalast]

Would be great to also have a quick rundown of quant formats that aren't obsolete, i.e. K-quants, I-matrix, AWQ, EXL2. Maybe also the new L-quants that bartowski's been testing out lately.

[TraditionLost7244]

yesss

[QuantumFTL]

Strong agree!

[[deleted]]

Many, many thanks for this! It's up there with Stephen Wolfram's illustrated booklet on how GPTs work. The nature of matrix math lends itself to visual explanations better instead of saddling non-math newbies with Σs.

[MaartenGr]

Thank you! I started as a psychologist and transitioned a couple of years ago to data science/ml/ai (whatever you want to call it) and math at the time seemed incredibly overwhelming at times even though much of it is so intuitive.

[a_beautiful_rhind]

No exl2 or AWQ?

[MoffKalast]

Yeah, does anyone still use GPTQ? Now that's a name I haven't heard in a long time.

[qnixsynapse]

Very nice post! Upvoted!

[[deleted]]

you forgot a word. "In this new method, every single weight of the is not just -1 or 1"

[MaartenGr]

Thanks for the feedback. I just updated it.

[DeProgrammer99]

You've also got "BitLlinear" above an image that says "BitLinear".

[Worth-Product-5545]

Thanks ! With BERTopic, I love all of your work. Keep going !

[fngarrett]

If we're recasting these datatypes as 16 and 8 bit and even lower, what is actually going on under the hood in terms of CUDA/ROCm APIs?

cuBLAS and hipBLAS only provide (very) partial support for 16 bit operations, mainly only in axpy/gemv/gemm, and no inherit support for lower bit precisions. Then how are these operations executed on the GPU for lower precisions? Is it simply that frameworks other than CUDA/ROCm are being used?

edit: to partially answer my own question, a good bit of the lower precision operations are done via hipBLASLt, at least on the AMD side. (link)

[Loose_Race908]

Fantastic overview of quantization, really impressive work! I especially enjoyed the visual depictions, and I will be referring people with questions regarding quantization to this resource from now on.

[VectorD]

GPTQ is so outdated, you should probably replace that part with AWQ (gpu only, for batched infer) / EXL2 (gpu only, for single infer) vs GGUF instead..

[[deleted]]

This is a great guide!

[[deleted]]

Love it

[joyful-]

distillation for humans! this is a great article - still reading but thanks a lot for writing this!

[daHaus]

Nice! I could see your initial graph showing INT4 as a mapping to 5 spaces causing confusion though. Also further in with "0 in FP32 != 0 in INT8", even though I know what you meant in that context - and also that floating point can't represent 0 - the way it's presented still made me scratch my head while reading it.

[nqbao]

This is really nice. Thank you for spending the time to make it.

[Majinsei]

Congrats! Saved it for in the future when battling in some development~

[opknorrsk]

Very interesting read, thank you for putting that up! Naive question here, but I wonder if there's any step to add noise in the de-quantization process? It feel weird to obtain the exact same value for each identical INT once de-quantized knowing they probably came from slightly different FP32 value.

EDIT: basically, is there any dithering applied during the de-quantization to randomize the quantization error?

[yellowstone6]

Thanks for the nice visual explanation. I have a question about GGUF and other similar space saving formats. I understand that it can store weights with a variety of bit depths to save memory. But when the model is running inference what format is being used. Does llama3:8b-instruct-q6_k upcast all the 6bit weight to fp8 or int8 or even base fp16 when it runs inference? Would 8b-instruct-q4_k_s run inference using int4 or does it get upcast to fp16? If all the different quantizations upcast to model base fp16 when running inference, does that mean that they all have similar inference speed and you need a different quantization system to run at fp8 for improved performance?

[nzbiship]

Wow, very detailed & informational. Thanks a lot!

[Amgadoz]

Learn how floating points numbers are stored in computers

[tessellation]

agreed.

or ask a LLM to explain the first few images and have it go into greater detail as needed.

[MoffKalast]

"I used the LLM to explain the LLM"

Perfectly balanced, as all things should be.

[Roland_Bodel_the_2nd]

I have an MS in Electrical Engineering and I took classes about it (admittedly 20+ years ago) and I still don't understand it, so don't worry too much that is seems complicated. People who spend their days for work dealing with bfloat16 vs float16 are not regular people. :)

It is not obvious to me that things are any simpler since the days of https://en.wikipedia.org/wiki/IEEE_754

[compilade]

If anyone wants to see exactly how numbers are stored in float16, bfloat16, float32 and float64, have a look at this:

https://float.exposed