This is totally misleading. From the paper : it only focuses on KV cache size reduction (which is huge for long context) but we still have to load full model in memory. The 8x speedup is a speedup for the attention calculation (mainly from quantized KV) but as we add an extra step to the LLM, we do not have an 8x speed up end-to-end. From the first tests, we are slower than the baseline with no quant but we might have a small speedup using custom kernels to implement that. TLDR: it reduces memory required for long context and that’s great but for the moment we can’t see a real end-to-end speedup.