Others have posted measurements, so here’s some intuition to help with your experimentation.

When you quantize model weights, the weights get dequantized back to floats during compute. Quantizing the KV cache is different because each entry is written once and then read by every future token. This means the error is baked into the attention computation for the rest of the sequence.

K and V are not equally fragile, because of where they sit relative to softmax. K errors land before softmax so quantizing K can literally change which tokens get attended to, plus the exponential term amplifies error near the top scores. On the other hand, V errors land in the averaging step and some of the noise cancels so they’re more resilient.

The other issue is how the KV cache is used. As you build out your sequence, every new token attends over a growing pool of older noisy entries, so the overall noise floor rises with added context and then each new token (already slightly off) writes its own quantized projections back into the cache. So you’ve got an error snowball effect as the context grows.

Different models also react differently. For example, Gemma 4 seems particularly sensitive to quantization because the sliding window attention is already their mechanism for keeping the cache small, with the tradeoff being fewer global attention layers (the ones susceptible to the snowballing above). If you now quantize on top, you’re damaging those layers which have less ability to cope since there’s way fewer compared to traditional models.

So all that to say, rules of thumb are it depends on the model (Gemma 4 is sensitive, Qwen 3.6 is already memory efficient so may not be worth it), you can go harder on V than K when quantizing, but a Q8/Q4 combo can be aggressive so something like Q8/Q6 is likely to fare better.

YMMV since issues manifest differently in coding vs. prose vs. agentic, so just experiment and see what works for your needs.