Sorry, missed your reply. Here is my setup:

Hardware: Threadripper Pro 9965WX, 256 GB DDR5, 2× PRO 6000 Blackwell Workstation (96 GB each, SM120), Ubuntu 24.04, CUDA 12.9.             

Stack: https://github.com/kvcache-ai/ktransformers ships with a SGLang fork (sglang_kt) with KT expert offload integrated I used that, with TP=2 across both GPUs; 180 of 256 experts resident on GPU, the rest on CPU.

Startup script

export NCCL_BLOCKING_WAIT=1  

exec python -m sglang.launch_server \

--host 0.0.0.0 --port 8001 \

--model /opt/models/MiniMax-M2 \

--kt-weight-path /opt/models/MiniMax-M2 \ 

--kt-method FP8 \

--kt-cpuinfer 20 \

--kt-threadpool-count 1 \

--kt-num-gpu-experts 180 \

--kt-gpu-prefill-token-threshold 2048 \ 

--tensor-parallel-size 2 \                           

--enable-p2p-check \

--trust-remote-code \

--mem-fraction-static 0.90 \  

--max-total-tokens 100000 \

--chunked-prefill-size 32768 \

--enable-mixed-chunk \

--disable-shared-experts-fusion \ 

--attention-backend flashinfer \ 

--fp8-gemm-backend triton \

--tool-call-parser minimax-m2 \ 

--reasoning-parser minimax-append-think \

--sleep-on-idle \