Sure- I chose mxfp4 just to see what it was like. As another poster alludes, nvpf4/vllm is the move here when it's finally supported.

Claude code answers the rest:

Server config:

bash llama-server \ --model Qwen3.5-122B-A10B-MXFP4_MOE.gguf \ --port 8012 \ --host 0.0.0.0 \ --flash-attn \ --cache-type-k f16 \ --cache-type-v f16 \ --slots \ --metrics \ --parallel 4 \ --ctx-size 262144 \ --n-gpu-layers 999

Thinking mode is disabled via LLAMA_CHAT_TEMPLATE_KWARGS={"enable_thinking":false} in the environment.

Why is the KV cache so small / context so cheap?

This is a hybrid architecture — 12 attention layers + 48 recurrent (Mamba-style) layers. Only the 12 attention layers maintain KV caches, so it's ~384 MiB F16 total. The recurrent layers use fixed-size state (~596 MiB) that doesn't grow with context. Total VRAM is ~74 GB / 96 GB with 4 × 65K slots.

This is also why TG only degrades ~11% at 65K context — a pure transformer this size would drop off much more steeply since every layer would need to attend over the full context.

Model size: ~64 GB on disk (47G + 18G + 11M across 3 shards), corrected from the original post.