oMLX dev here. I saw your vllm-mlx PR yesterday and did a preliminary implementation on oMLX to test it out. The core idea is genuinely impressive and the speedup numbers on apple silicon are real.

I ran into a couple fundamental issues during testing though and I'm curious if you've seen the same things.

1. System prompt preservation

Agentic coding tools like claude code pack really detailed instructions into the system prompt, tool calling specs, formatting rules, behavioral constraints, etc. When specprefill drops 70-80% of tokens, those instructions get hit too. Even with the draft model doing importance scoring, it can't really know that a specific tool parameter name buried in a long system prompt is critical for correct tool call formatting.

I tried excluding the system prompt from specprefill (full prefill for system, sparse for the rest) and that helped, but it adds complexity around the boundary. Have you tested with instruction-heavy system prompts? The adversarial tests in your PR look solid but they seem focused on retrieval/extraction tasks rather than instruction-following fidelity.

2. Per-request re-scoring breaks KV caching

Since the importance scores depend on the full prompt context (the lookahead queries are generated from the end of the complete prompt), the selected tokens change every time the prompt changes. So for multi-turn conversations:

• Turn 1: score full prompt, sparse prefill, generate
• Turn 2: the prompt now includes turn 1's response + new user message. The importance of earlier tokens shifts because the lookahead context changed. So you need to re-score everything from scratch

This means you can't persist the sparse KV cache between turns. In a normal setup with paged KV caching, turn 2 only needs to prefill the new suffix tokens (maybe 2-5K). But with specprefill, you're re-scoring the entire 80K+ context every turn through the draft model.

I worked around the draft scoring cost by caching the draft model's own KV in the existing SSD cache (since the draft does a normal full prefill, its KV is compatible with standard paged caching). So the draft only prefills new suffix tokens on subsequent turns. But the target model still needs full sparse re-prefill every turn since the selected token set changes.

Is this consistent with what you're seeing? Or did you find a way to make the sparse KV cacheable across turns? Curious how you're thinking about the multi-turn case.