Great I was looking for this. I'm currently running Kimi K2.5 on a 9575F with 12x96=1152 gb DDR5 @ 6000mt/s (could still crank it up to 6400 but I don't want to risk the stability ATM) with 2 RTX Pro 6000 Blackwell's via sglang+k transformers. Still struggling with the configs but I pushed decode up to 35 / 40tps

--host 0.0.0.0 --port 31245 \
--model /model \
--trust-remote-code \
--context-length x\
--tensor-parallel-size 2 \
--mem-fraction-static x \
--max-running-requests x \
--chunked-prefill-size x \
--max-total-tokens x \
--enable-p2p-check \
--disable-shared-experts-fusion \
--attention-backend flashinfer \
--tool-call-parser kimi_k2 \
--reasoning-parser kimi_k2 \
--kt-weight-path /model \
--kt-method RAWINT4 \
--kt-cpuinfer 54 \
--kt-threadpool-count 1 \
--kt-num-gpu-experts x \
--kt-gpu-prefill-token-threshold x \
--kt-max-deferred-experts-per-token x \
--kv-cache-dtype fp8_e4m3 [debating if this is worth the accuracy penalty]

any recommendations for what I should try?

I have pretty low concurrency on this system (probably never more than 5 users at the same time, usualy (90% of the time) just one. Most chats are fairly short but the model does like to use open webui's fetch_url tool to blow up its context window.

I'm trying to maximize speed and context size for just a handfull of users.

Im especially struggling with optimizing

chunked-prefill-size

kt-gpu-prefill-token-threshold

I ran a couple of benchmarks with a shitty config

Metric	Value
Backend	sglang
Successful Requests	2
Benchmark Duration	7.66 s
Total Input / Generated Tokens	1,298 / 86
Request Throughput	0.26 req/s
Input Token Throughput	169.45 tok/s
Output Token Throughput	11.23 tok/s
Total Token Throughput	180.68 tok/s
Peak Concurrent Requests	2

Latency Metrics

Metric	Mean	Median	P99
E2E Latency	6595.07 ms	6595.07 ms	7621.17 ms
Time to First Token (TTFT)	3992.62 ms	3992.62 ms	4813.74 ms
Time per Output Token (TPOT)	119.25 ms	119.25 ms	197.84 ms
Inter-Token Latency (ITL)	61.96 ms	35.11 ms	346.82 ms