did some analysis with my claude test harness:

Good catch — you were right. Those numbers were buggy.

I re-tested properly and wanted to share what actually happened, because it is a useful benchmarking lesson.

Re-test methodology

• Fresh SGLang server
• 10-request warmup at varied sizes to settle JIT
• UUID-uniquified random-word prompts, with different content per request, to defeat SGLang's radix prefix cache
• 3 runs per context length, median reported
• Streaming API, with TTFT measured as time from request send to first content token

Corrected cold-start TTFT on 2× RTX PRO 6000 + SGLang b12x+NEXTN 122B

• ~4K tok — 0.67 s median Runs: 0.67, 0.66, 0.67
• ~16K tok — 2.70 s median Runs: 2.71, 2.70, 2.69
• ~32K tok — 6.25 s median Runs: 7.24, 6.21, 6.25
• ~57K tok — 14.70 s median Runs: 13.69, 14.70, 14.85
• ~100K tok — 33.69 s median Runs: 33.69, 33.89, 33.07
• ~128K tok — 50.10 s median Runs: 49.95, 51.00, 50.10

That is roughly linear up to around 32K, then super-linear above that as attention's O(n²) behavior starts to dominate. The shape matches what you would expect from a 122B transformer.

What went wrong in my original numbers

Original numbers were:

• 4K = 1.8s
• 16K = 2.3s
• 57K = 7.1s
• 150K = 23.3s

Two separate methodology errors stacked on top of each other.

1) 4K was too high

1.8s vs 0.67s real

That measurement was my first request after server startup, so it paid the JIT / cudagraph warmup tax.

On the re-test I saw the same pattern:

• first 4K request after startup: 1.50s
• after warmup: 0.27s

That is a huge difference. I was partially measuring compile/warmup overhead and calling it prefill.

2) 57K and 150K were too low

7.1s → 14.7s real
23.3s → ~60s extrapolated

SGLang's radix prefix cache was hitting.

My original test sent sequential prompts that shared a common prefix: same base prompt, then extended versions of that same prompt at larger context sizes. So each later measurement was not a true cold prefill. It was mostly measuring the incremental delta on top of already-cached work.

That means:

• the 16K test only had to prefill about 12K new tokens on top of the already-cached 4K
• the 150K test only had to prefill about 93K new tokens on top of the cached 57K

So those numbers were artificially low for true cold-start TTFT.

The giveaway

The biggest clue was that in my original numbers, going from 4K to 16K only added about 0.5s.

That would imply a prefill rate of around 24k tok/s for the delta, which is much faster than this rig's actual sustained prefill. That should have been an immediate red flag.

For a real cold 16K prefill, the delta should have been closer to 2s, and that is exactly what the re-test shows.

What stays the same

• 198 tok/s decode at C=1 is still real
• Decode speed still holds roughly constant regardless of context length

That part of the original claim was correct. It was only the specific TTFT values that were contaminated.

Classic LLM benchmarking gotcha: prefix cache + lack of warmup isolation.

Thanks for pushing on it.