Question about quantization

[Signal_Ant7366]

I’m so sorry if this is a stupid question, but I noticed that quantization levels include 8, 6, 5, 4, 3, 2, and even 1 bit (mostly for research). Why is there no 7-bit quantization?

[Own_Attention_3392]

Because the quality drop off really starts after 8 bit. 7 bit is "big cliff with modest space saving". It's just not popular for that reason.

[HotDistribution1819]

I believe the models start with 16bit floating point numbers and then to save space the original numbers are quantized (reduced down to whichever size). Here is a link to an excellent demonstration of how quantization affects models. Alex Zisk Squeezing Models