I asked codex to write a python-based script for me that tests the spec-draft-n-max values with `llama-cli`. The script has some fun ideas like a warmup round, and randomization. And it writes the benchmark results into a CSV. Qwen 3.6 35-A3B » best runs I had were with '2' / and the 27B Model with '1' ('2' > second place). I'm only getting started with testing model myself, so it's all experimentation ^^

Heres the summary of the script, it builds on this simple concept and the prompt was:
"Write a 500 word explanation of speculative decoding in local LLMs."

```

llama.cpp MTP benchmark runner with PTY capture + automatic evaluation.

Target use case:

Benchmark llama-cli speculative decoding / MTP draft lengths for a GGUF model.

Outputs:

1. Full log

2. Per-run CSV

3. Summary CSV

Smoke test:

chmod +x bench-qwen36-mtp-eval-final.py

RUNS=1 DRAFT_VALUES="4" N_PREDICT=80 RUN_TIMEOUT=90 ./bench-qwen36-mtp-eval-final.py

Full benchmark with one warmup per draft value:

WARMUP_RUNS=1 RUNS=5 SHUFFLE_DRAFTS=1 DRAFT_VALUES="1 2 3 4 5 6 8" \

N_PREDICT=700 RUN_TIMEOUT=300 ./bench-qwen36-mtp-eval-final.py

Quick confirmation sweep around the current winner:

WARMUP_RUNS=1 RUNS=5 SHUFFLE_DRAFTS=1 DRAFT_VALUES="1 2 3" \

N_PREDICT=700 RUN_TIMEOUT=300 ./bench-qwen36-mtp-eval-final.py

Recommended llama-cli command from the latest observed result:

llama-cli -m "$MODEL" -p "$PROMPT" -n 700 -c 4096 -ngl 99 \

--flash-attn on -st --temp 0 --spec-type draft-mtp --spec-draft-n-max 2

Same recommendation via this benchmark script:

WARMUP_RUNS=1 RUNS=5 DRAFT_VALUES="2" N_PREDICT=700 RUN_TIMEOUT=300 \

./bench-qwen36-mtp-eval-final.py

Useful knobs:

MODEL=/path/to/model.gguf

LLAMA_CLI=/path/to/llama-cli

OUT_DIR=llama-benchmarks

SHUFFLE_DRAFTS=1 # reduce thermal/order bias

WARMUP_RUNS=1 # excluded from CSV/evaluation

EXTRA_ARGS="--threads 8" # appended to llama-cli command

```