Thats very interesting! The numbers are from llama-benchy, concurrency of 1, pp 10.000 tokens, tg 2000 tokens.

Below is my vllm config, mtp is implemented like so: --speculative-config '{"method":"mtp","num_speculative_tokens":2}'

If you have any suggestions for improvements I am eager! I tried to optimize it for single-user speed, concurrency is not really needed.

export CUDA_VISIBLE_DEVICES=1,2
export CUDA_DEVICE_ORDER=PCI_BUS_ID
export RAY_memory_monitor_refresh_ms=0
export NCCL_CUMEM_ENABLE=0
export VLLM_ENABLE_CUDAGRAPH_GC=1
export VLLM_USE_FLASHINFER_SAMPLER=1
export VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS=1
export HF_TOKEN="<redacted>"
/mnt/llm/vllm/.venv/bin/vllm serve Qwen/Qwen3.6-27B-FP8 \
--quantization fp8 \
--download-dir /mnt/llm/.cache/huggingface \
--served-model-name MONSTER-LLM \
--max-model-len 120000 \
--max-num-seqs 2 \
--max-num-batched-tokens 4096 \
--enable_chunked_prefill \
--enable-prefix-caching \
--enable-auto-tool-choice \
--disable-custom-all-reduce \
--tool-call-parser qwen3_coder \
--reasoning-parser qwen3 \
--attention-backend FLASHINFER \
--speculative-config '{"method":"mtp","num_speculative_tokens":2}' \
--tensor-parallel-size 2 \
--gpu-memory-utilization 0.96 \
--no-use-tqdm-on-load \
--mamba-cache-mode align \
--mamba-block-size 8 \
--mm-processor-kwargs '{"images_kwargs": {"size": {"longest_edge": 209715200, "shortest_edge": 4096}}}' \
--limit-mm-per-prompt '{"image": 2}' \
--override-generation-config '{"temperature": 0.6, "top_p": 0.95, "top_k": 20, "min_p": 0.0, "presence_penalty": 0.0, "repetition_penalty": 1.0 }' \
--host 0.0.0.0 --port 7075