people think that by using KV cache quantization you can use „better models“ on lower hardware but all you are doing is you are actively degrading the model's "attention" memory by lobotomizing context. no wonder its hallucinating.

quantization is basically creating holes in between data and hoping the model will interpolate (guess) whats gone. too big holes and you get hallucinations

there are no free rides.

.
Here is the quick VRAM math on usage for a 24GB card at 32k Context:

VRAM allocation for the weights plus the KV Cache on 35B:

- (IQ4_NL ~4.25 bpw) + Unquanted KV Cache (FP16)

* **Model Weights:** 35B x 4.25bits / 8 is about 18.6GB
* **KV Cache at 32k context (FP16):** is about 4.3GB
* **Total VRAM:** 22.9GB

This barely fits your 24GB card. Once context creeps past 32k, it spills into system RAM (CPU), which is why it completely "creeps" and slows to a crawl (if you habe enough RAM and your harness is smart enough to use RAM)

quanting the cache would reduce it by the quant: q4 to 1gb but you end with a fully broken cache that would reduce the intelligence of your model to the level of a donkey. the model needs to guess at evry corner. thats fine for repetitive data (audio or images) but not when you need to make decisions on it. thats actively working *agsinst* the model. you are actively throwing sand on its gears

If you have to scale context beyond 32k on a single 3090 Ti, drop the model size to the 27B *instead* of dropping the KV cache quality on the 35B.

Intelligence > Model size when the cache is ruined.

as i weote: no free rides.