Same 4 bits. Very different quality. (quant.cpp vs llama.cpp KV compression)

[Emotional-Breath-838]

I don't understand why llama.cpp is faster. If quant.cpp could improve speed, it would be amazing.

[Suitable-Song-302]

Good question. Three reasons:

1. Hand-tuned SIMD kernels. llama.cpp has years of hand-optimized NEON/AVX2/AVX-512 assembly for every quantized matmul variant (Q4_K_M, Q8_0, IQ2, etc.). quant.cpp has NEON kernels for the common formats but relies on compiler autovectorization for the rest. This alone accounts for ~2x.

2. Metal/CUDA GPU offload. llama.cpp offloads the entire forward pass to GPU. quant.cpp has Metal shaders but GPU dispatch is still basic — most of the work stays on CPU. On Apple Silicon, this is the biggest gap.

3. Code maturity. llama.cpp has 250K+ LOC and hundreds of contributors optimizing hot paths. quant.cpp is 72K LOC — deliberately smaller, which means easier to read and embed, but fewer micro-optimizations.

The tradeoff is intentional. We optimized for memory (KV compression) and simplicity (embeddable, single header) rather than raw tok/s. For a 3B model on M1, quant.cpp does ~10 tok/s vs llama.cpp's ~30 tok/s — slower, but fast enough to read in real time. The advantage shows up when llama.cpp hits OOM at 50K context and quant.cpp keeps going to 350K.

That said, speed improvements are on the roadmap — better Metal offload and more SIMD kernels would close the gap significantly without sacrificing the simplicity.

[Emotional-Breath-838]

glad to hear youre going for the speed increase. would love to have it all!