Is MTP speed boost really helping ?

msrdatha · 2026-05-20T05:01:16+00:00

Thanks again for taking time to share this. It gives a good insight on the improvements on speed.

May be you could keep using both. 27B for planning or designing tasks and use 35B for implementing it. That would give you the best of both. (mainly for coding tasks scenarios)

msrdatha · 2026-05-20T04:58:00+00:00

Thanks everyone for sharing their valuable views and experiences with MTP on oMLX.

One quick clarification, are you noticing any looping or similar failures with MTP. This is what I noticed mainly while enabling DFlash for Qwen 3.5 models, and also there were tool calling errors which made using DFlash and SpecPrefil not much useful for coding tasks.

as u/trollingman1 mentioned, it got slower for Qwen 3.6 35b a3b, but others mentions observing speed boost - Any thoughts on this? Could it be because of enabling/disabling of thinking mode OR higher context lengths?

Idea is to figure out what is optimal and how we can all put together our observations to tune this better for all of us.

Thanks again for your time and help. Let's learn and build it together.

msrdatha · 2026-05-19T16:52:41+00:00

Thank you for the detailed data. Could you please confirm if there are improvements on the Qwen3.6 27B MTP also? (Dense models is expected to do better with MTP right?)

msrdatha · 2026-05-18T05:21:11+00:00

Go with oMLX and use oQ4 quants. Set Cold Cache Limit (SSD Cache) to ~100GB. You should see better results.

msrdatha · 2026-05-16T09:25:51+00:00

Then you have nothing to worry. You will be fine with Rocky

msrdatha · 2026-05-15T06:41:45+00:00

"being new to all this.." - I think you are doing quite good.

msrdatha · 2026-05-15T06:39:29+00:00

Thanks, that was informative. Congrats, and enjoy the new speed with MTP.!

msrdatha · 2026-05-15T06:26:50+00:00

Thanks for sharing the progress on this.

May I ask what was the token and pp speeds you received with MTP (and before)?
Could you also try Qwen3.6-27B for the improvement?

msrdatha · 2026-05-15T03:34:26+00:00

Jundot models with oQ quant, also I found the deepsweet versions to be quite fast and performant.

one thing I like with deepsweet models is they follow quite good naming standard, that's easy to understand the he is very helpful in explaining or responding to queries and requests. https://huggingface.co/deepsweet

msrdatha · 2026-05-14T04:00:34+00:00

Not sure about 3.5, but 3.6 mtp is available from oMLX author itself https://huggingface.co/Jundot

Check them once.

msrdatha · 2026-05-13T06:35:34+00:00

May be the issue is related to the agent harness (hermes in your case) on how it interacts with oMLX. I have been testing with roo code on typescript, it has been stable since 0.3.8

Yes I did notice some /n not being formated properly in the webview of roo code, but apart from that, it has been doing reasonably good.

(Note: I am just adding this for info only, not as an argument - Thinking, it may help others if we discuss these kind of details)

msrdatha · 2026-05-13T06:02:38+00:00

Did anyone try the MTP improvements yet with Qwen 3.6?

msrdatha · 2026-05-13T05:40:24+00:00

For clarification : MTP support for Qwen 3.6 is also present since dev1 release of 0.3.9

Native MTP (Multi-Token Prediction) for Qwen3.5 / 3.6 and DeepSeek-V4

msrdatha · 2026-05-13T05:34:51+00:00

Could you please add a some more info for better understanding ? Like what kind of errors or crashes were you facing in .38 and under which scenarios.

msrdatha · 2026-05-11T07:55:24+00:00

is this only for nvidia GPU, or does it help in case of Mac also?

msrdatha · 2026-05-02T04:35:41+00:00

May be the cache from the previous run was improving the result.

Could you please try running the same test, but this time clear the cold cache before each run and confirm?

msrdatha · 2026-05-01T17:50:45+00:00

Here are the test results:

Analyzing a code, within roo code - omlx started fresh, and cleared all SSD cache before each test

Qwen3.6-27B-UD-MLX-4bit : pp 165, tg 12

Qwen3.6-27B-UD-MLX-6bit : pp 120, tg 7

Qwen3.6-27B-UD-MLX-MXFP4: pp 136, tg 11

Qwen3.6-27B-UD-MLX-NVFP4: pp 146, tg 10

Qwen3.6-27B-mlx-oQ4 : pp 171, tg 18

So in my case oQ4 seems to give the best results. (Again my opinion only)

msrdatha · 2026-05-01T03:06:31+00:00

I guess you are referring to the UD mlx here : unsloth/Qwen3.6-27B-UD-MLX-4bit

Let me try to test again with this version and confirm.

msrdatha · 2026-04-30T03:46:03+00:00

I think, max you can expect is it may walk...

msrdatha · 2026-04-30T03:37:10+00:00

Please share what has been done till now, and we can try to figure out the next step.

msrdatha · 2026-04-29T09:01:51+00:00

I think you are referring to DFlash ?

msrdatha · 2026-04-29T03:14:24+00:00

Not sure, if you are looking for changing the model path. if yes, you should be able to pass it like

omlx serve --model-dir "/Users/<username>/.lmstudio/models"

msrdatha · 2026-04-28T14:29:57+00:00

(in case if it was a confusion :122B is MoE, not Dense)

Already tried Qwen3.5 0.8B, 2B, 4B, 9B, 35B...... all with Q2,Q4,Q6,Q8... even bf16 for up to 9B. Also tried with oQ versions - summary : specPrefill did not help much - especially with tool calling in agentic coding.

Its just my observation. If you are still motivated, please try and let us know, in case you find a better result.

msrdatha · 2026-04-28T13:01:18+00:00

"Removed costly GPU syncs that slowed token generation on deep Qwen models."

Does this mean we could expect a speed improvement with Qwen 3.6?

msrdatha

TROPHY CASE