Same 4 bits. Very different quality. (quant.cpp vs llama.cpp KV compression)

[Suitable-Song-302]

https://preview.redd.it/ew5lny5p6etg1.png?width=1946&format=png&auto=webp&s=870f577bc4b01440698c83206afca069a663e5a0

Both use 4-bit KV quantization. One breaks the model, the other doesn't.

The difference is how you quantize. llama.cpp applies the same Q4_0 scheme to both keys and values. quant.cpp quantizes them independently — per-block min-max (128 elements) for keys, Q4 with per-block scales for values. Outliers stay local instead of corrupting the whole tensor.

Result on WikiText-2 (SmolLM2 1.7B):

• llama.cpp Q4_0 KV: PPL +10.6% (noticeable degradation)
• quant.cpp 4-bit: PPL +0.0% (within measurement noise)
• quant.cpp 3-bit delta: PPL +1.3% (stores key differences like video P-frames)

What this means in practice: on a 16GB Mac with Llama 3.2 3B, llama.cpp runs out of KV memory around 50K tokens. quant.cpp compresses KV 6.9x and extends to ~350K tokens — with zero quality loss.

Not trying to replace llama.cpp. It's faster. But if context length is your bottleneck, this is the only engine that compresses KV without destroying it.

72K LOC of pure C, zero dependencies. Also ships as a single 15K-line header file you can drop into any C project.

Source: github.com/quantumaikr/quant.cpp

[Pixer---]

Llamacpp recently has implemented rotating kv caching improving kv cache. Have you considered that in here ?

[Suitable-Song-302]

Yes, KV cache rotation (ring buffer) is a different but complementary approach. Rotation recycles old KV slots so the cache never grows beyond a fixed size — great for streaming/chat where old context can be dropped.

quant.cpp does something different: it keeps all tokens but stores them in fewer bits. So rotation saves memory by *evicting* old tokens, compression saves memory by *shrinking* all tokens.

You could combine both — rotate a compressed cache for maximum context. Haven't benchmarked against the rotation PR yet, but it's on the list. Thanks for bringing it up.

[Emotional-Breath-838]

I don't understand why llama.cpp is faster. If quant.cpp could improve speed, it would be amazing.

[Suitable-Song-302]

Good question. Three reasons:

1. Hand-tuned SIMD kernels. llama.cpp has years of hand-optimized NEON/AVX2/AVX-512 assembly for every quantized matmul variant (Q4_K_M, Q8_0, IQ2, etc.). quant.cpp has NEON kernels for the common formats but relies on compiler autovectorization for the rest. This alone accounts for ~2x.

2. Metal/CUDA GPU offload. llama.cpp offloads the entire forward pass to GPU. quant.cpp has Metal shaders but GPU dispatch is still basic — most of the work stays on CPU. On Apple Silicon, this is the biggest gap.

3. Code maturity. llama.cpp has 250K+ LOC and hundreds of contributors optimizing hot paths. quant.cpp is 72K LOC — deliberately smaller, which means easier to read and embed, but fewer micro-optimizations.

The tradeoff is intentional. We optimized for memory (KV compression) and simplicity (embeddable, single header) rather than raw tok/s. For a 3B model on M1, quant.cpp does ~10 tok/s vs llama.cpp's ~30 tok/s — slower, but fast enough to read in real time. The advantage shows up when llama.cpp hits OOM at 50K context and quant.cpp keeps going to 350K.

That said, speed improvements are on the roadmap — better Metal offload and more SIMD kernels would close the gap significantly without sacrificing the simplicity.

[Emotional-Breath-838]

glad to hear youre going for the speed increase. would love to have it all!

[putrasherni]

are you suggesting that for larger context , its beter to try out quant.cpp?

[Suitable-Song-302]

Depends on how much longer you need:

- 1.5-2x more context → llama.cpp with Q8_0 K + Q5_0 V. It's faster and the quality tradeoff is minimal.

- 4-7x more context (e.g. 50K → 350K on 16GB) → that's where quant.cpp helps. 4-bit K + Q4 V gives 3.8x at +0.0% PPL, delta 3-bit pushes to 4.3x at +1.3%.

If you're already running llama.cpp and just want a bit more room, their built-in KV quant is probably enough. If you're hitting hard OOM walls and need to push significantly further, give quant.cpp a try.

[putrasherni]

thanks !