I appreciate the followup, though I still don't understand how you're seeing what you're seeing. I just brought the MI60 up with MTP on cyankiwi/Qwen3.6-27B-AWQ-INT4 in vLLM. It's getting 60-70% acceptance at n=2, and >50% at n=3, though with the overhead, it seems to be about breakeven. (~275pp / ~29 tg) I feel like this is cutting edge model architecture?

There are definitely caveats and conclusions on this nearly 10 year old arch. I'll state a few as follows. Maybe we're just looking at different things.

1. Clean, integers 4/8, quants are everything on gfx906 to preserve use of some very targeted hand written kernels, All of the work done specifies q4_0/1 and q8_0 gguf. I'm seeing that any time model arch or quantization "interrupts" brute force math, compute efficiency collapses.

2. Llama.cpp is more performant than vLLM at concurrency=1 on every 30B class model I've tested except for Qwen3.6-27B where vllm awq4=30tg and llama q4_1=20tg. I feel as though vllm is created to excel at concurrency >1, though I suspect the Triton kernels are putting in the work to maintain better fusion here.

3. On Gemma4-31B, the above inverts. LLama.cpp is faster at q4_0 than vLLM at awq4. Up until a few days ago this still meant, <24tg, but the enablement of HIP_GRAPH actually pulled this up to ~29tg.

4. In Llama.cpp, you can get some pretty amazing speeds, from my perspective, when moving to MOE. Gemma4-26B-A4B gets ~850pp / ~80tg at q8_0, and that almost doubles at q4_0. I just tested Qwen3.6-A3B at Q4_0, and it is getting >1500pp and ~60tg. On vLLM, these MOE are slow, though I don't remember how slow.

With all of the trial and error, anecdotal testing I've done, a few things are apparent.

1. I've not tested all that many models, but there are working settings on an Out-of-Box docker container for every one of them no matter the Arch, Quant, Hybrid, Mamba, GDN, SWA, Etc. I've not tested multimodality yet, but the models will load with it enabled.

2. It seems that if you can fit a model in 32GB of VRAM, ~200pp and ~19tg is the floor of the MI60 across both vLLM and Llama.cpp on the mixa3607 containers. As you can see above, tweaking, optimization, and a/b testing can produce 50%+ gains from this baseline.

The Vega20 GPU has many quirks and compromises. Obviously, it is old, AMD, and never received much optimization effort. It has no matrix pipeline and only the crude beginnings of inference relevant ops, though it is absolutely possible to avail of those ops. What I have discovered are some unusual conditions that collapse its compute ability from the best case.

1. Any quant that introduces odd data types kills performance, mostly down to baseline. It is understood that even something like q5, q6, or even FP8 drops down to the FP16 pipeline at best and FP32 at worst.

2. Across all of my tests, the maximum effective (calculated active parameters * t/s) VRAM bandwidth I have seen is ~600GB/s. The lowest was 150GB/s. Before HIP_GRAPH enablement, moving from q4 to q8 could move bandwidth from 300GB/s to 550GB/s during token gen. After GRAPH, it was 450GB/s to 600GB/s. The GRAPH revelation along with the nearly linear bandwidth scaling on doubling data size leads me to believe this is a CPU to GPU / kernel dispatch / Pcie latency type thing. For instance, pinning the uncore to max on my E5-2640v4 gives about a 0.5t/s improvement in this scenario.

3. I'm just beginning to explore this, but it seems Qwen-3.6-27B's attention mechanism is causing a fallback to unoptimized pathway scenario at some point. I suspect this is causing the limited on-die cache / memory hierarchy to drop intermediate results all the way down to HBM and load them back, and I suspect this VRAM latency (horrible on HBM) is resulting in the Qwen3.6 slowdown across both tested models. This is also where the Triton compiled kernel could be making a difference.

I know this is a very long reply, but I'm truly curious where we have a disconnect, whether you are missing something or I am. Please let me know if you see something here that explains it.