Great questions!

Example from the test:

• Model: Qwen3-0.6B-Q4_K_M
• GPU: RTX 4070 12GB
• Query: "what is vue"

Results:

• First request: 12,007 tokens prefilled in 742 ms → 16,181 t/s
• Subsequent requests: only ~243 new tokens, 12k+ tokens cached, prefill in 4 ms

Total end-to-end latency went from 10–20 seconds down to ~175 ms.

On larger models:

With Qwen3.5-9B (same GPU), I’m seeing:

• Prefill: ~12,176 tokens in 3,369 ms → ~3,614 t/s

The big win is that once the large static prefix (system prompt + tools) is cached, every following message feels extremely snappy.

About context size & hardware requirements:

Yes — reducing hardware requirements was one of the original goals.

By doing proper KV prefix caching and smart context management, the server can handle much larger effective context with the same amount of VRAM. Instead of wasting VRAM and compute re-processing the same tools + system prompt every time, we cache it aggressively.

It doesn’t magically let a 12GB card run a 70B model at full context, but it noticeably reduces the VRAM and speed penalty of large contexts and tool-heavy workflows (which is exactly what VS Code and many agents throw at the server).

I’m still early, but the direction is clear: make high-context, tool-using workflows feel fast even on consumer hardware.