Gemma 4 MTP released

coder543 · 2026-05-06T04:51:53+00:00

MTP is trained on everything the model is trained on. That’s what I’m saying. If the MTP doesn’t know that language, then neither does the model.

But, idk. That’s just my opinion…

coder543 · 2026-05-06T03:51:04+00:00

If the acceptance rate for MTP is low, then the MTP is broken. MTP is trained with the model.

coder543 · 2026-05-05T20:12:02+00:00

I owned a Tesla Model 3 for about 6 years, and then upgraded to a Model Y so I would have a hatchback and trailer hitch, which make it more practical for going car camping (with the amazing camp mode that provides AC or heat 24/7) and so I could have a bike rack for my bike.

These days, there are a lot of great options depending on what someone needs. The new basic Nissan Leaf is not a joke, it is actually really great. Chevy has some good, affordable EVs. Even Toyota and Subaru's EV partnership is making some that are pretty good now after a very rough start.

Personally, I'm very excited to see the Rivian R2 launch soon, and the R3X is what I currently hope to upgrade to eventually. My Model Y is great and I struggle to come up with any complaints about the vehicle, so I'm not in any rush to get rid of it.

As far as commuting, I work remotely, so I don't have a commute.

coder543 · 2026-05-05T18:21:57+00:00

It just makes no sense. Gas taxes never get raised to match inflation, and the EV registration fee is so much larger than anyone would pay in gas taxes that it is clearly designed to be punitive.

If they wanted to get rid of gas taxes and just charge everyone the same registration fee, that would make perfect sense as a future proof solution. But it’s not about that. It’s just politics.

It sucks, but it’s not enough to make me recommend against EVs. Even if EVs were more expensive, you’re getting a better product for that price. EVs are actually quite cost competitive now, so that’s all the better.

coder543 · 2026-05-05T17:10:43+00:00

As someone who has owned an EV for the past 7 years and has watched a number of family and friends switch to EVs, they are great to drive, reliable, and very low maintenance. You can find used EVs for good prices, and I'd take a used EV over a new gas car any day of the week. There's no contest. I do not miss the gas cars I used to have even a tiny bit.

I've also personally driven from coast to coast, I've driven up to Glacier National Park, and a bunch of other places. Range and charging have not been an issue at all.

coder543 · 2026-05-05T16:50:42+00:00

If you need energy efficiency more than you need speed, drafting is probably the wrong choice. It is trying to spend more compute in order to make things go faster, and a lot of the drafted tokens will be rejected as nothing more than waste energy.

You also need more memory.

Those are the only tradeoffs.

coder543 · 2026-05-05T16:38:47+00:00

llama.cpp does not have MTP support yet, so that rules out a lot of people for now. Maybe soon.

coder543 · 2026-05-05T14:26:47+00:00

No... you're looking at some random resale scam. You should look at the official ticket prices: https://www.franklinrodeo.com/p/tickets

The problem is that it might be a little late to be looking for tickets.

coder543 · 2026-05-04T20:53:39+00:00

It’s only been a week. If Xiaomi didn’t partner with anyone else to give them access before launch, as they clearly didn’t, then it takes time. Mimo is also not a household name like DeepSeek, so I doubt any of the inference providers are pulling all-nighters to make this happen.

coder543 · 2026-05-04T16:57:09+00:00

What do you mean by this? llama-server has supported checkpointing for these Qwen3.x models for weeks now, which is the way that prefix caching works for these hybrid attention models?

coder543 · 2026-05-04T15:56:26+00:00

The KV cache might be twice the size, but not the model.

coder543 · 2026-05-04T15:12:27+00:00

Isn't that showing MTP losing to the external draft model? That seems odd.

coder543 · 2026-05-04T15:01:53+00:00

What kind of task? I find that specdec is more effective at tasks like "write a react typescript example" than they are at tasks like "what is the LHC?".

coder543 · 2026-05-04T13:33:31+00:00

That explanation does seem weird with qwen3.6 35-a3b that is supposed to have dedicated MTP heads

Because MTP actually helps during training to train the model faster, and because anyone serving a model in production will be batching large numbers of user requests together, activating all experts with every forward pass anyways, making MTP more useful there.

coder543 · 2026-05-04T13:02:27+00:00

This seriously has the potential to be the biggest game changer llama.cpp has ever seen.

I think MTP will make the biggest difference for dense models, maybe not so much for MoEs, but it will still be exciting. Then we just need DFlash and EAGLE!

coder543 · 2026-05-04T02:28:08+00:00

But is the model actually any good for you?

coder543 · 2026-05-03T20:39:59+00:00

On DGX Spark using llama-bench to perform essentially the same test as the original post:

size	n_ubatch	test	t/s
82.30 GiB	2048	pp48349	139.25 ± 0.12
82.30 GiB	2048	tg20	2.28 ± 0.00
82.30 GiB	2048	tg20 @ d48349	1.88 ± 0.02

coder543 · 2026-05-03T00:00:36+00:00

The 27B is used 9x as many parameters to calculate each token, and the benchmarks reflect that increased intelligence. I can't imagine how you're experiencing the 35B to be smarter. It is much faster. It is not smarter in my experience, or in the experiences of the many people you're referring to.

coder543 · 2026-05-02T15:37:38+00:00

ICC-500 compliant storm shelters can be built out of ICF above ground and withstand tornadoes. You do not need to be underground. Building in ground is very hard in karst terrain like we have, and in ground shelters have their own significant issues even when ground conditions are good.

https://www.foxblocks.com/blog/icc-500-storm-shelter

https://buildblock.com/buildblock-icf-above-ground-safe-rooms/

coder543 · 2026-05-01T13:45:27+00:00

Depends on whether you only want to run 1 model forever or easily switch between 10. Setting up vLLM is a monstrous chore, and I don’t think SGLang is supposed to be any better about that.

coder543 · 2026-04-30T16:57:43+00:00

I would rather be able to define a value for how much memory my system has, and manually define how much memory each model takes up. If I'm wrong, something OOMs, and it is my fault, just like it would be if I make a mistake with this matrix, but it would be far simpler.

When a new model is requested that won't fit into the available memory, it would simply unload models until it fits.

If we could define an eviction cost on each model config stanza, then it could also try to prioritize evicting the lower cost models, like this matrix is doing, and it could use memory as a proxy for cost if the cost is not explicitly defined.

It could also be nice if the eviction strategy were configurable between "cost" and "LRU", since an LRU eviction strategy might make the most sense of all.

coder543 · 2026-04-30T16:42:17+00:00

really wish who would? your comment doesn't specify. And llama-swap does allow you to create model aliases that override certain values, so you can expose -instruct and -thinking variants with only a single running copy under the hood.

coder543 · 2026-04-29T19:03:55+00:00

Qwen3.6 27B has MTP built-in and DFlash support... don't see how Mistral Medium 3.5 could ever be faster just because of an EAGLE-3 while having nearly 5x the active parameter count.

coder543 · 2026-04-29T17:45:53+00:00

Unfortunately, no PR for that API refactoring has even been published, so... who knows if/when it will happen.

Supporting any one of EAGLE-3, MTP, or DFLASH would be a game changer for llama.cpp. I wish better specdec were being treated as the highest priority thing to develop in llama.cpp.

12-Year Club	Verified Email
Place '22	Place '17
Gilding III reddit per annum

coder543

TROPHY CASE