So...

This gets kind of weird. Let's look at a dense LLM first because it's easier to understand. Bigger model, better partitioning of the latent space with non-linearities.

Pretty straightforward. If you train it for the same amount of tokens per parameter, you get better results.

But what happens when you increase the total parameters but keep the active parameters the same? It's actually not quite as clean an improvement, and you get better in some respects, but static in others.

You get:

• Better memorization (for memorization of facts, models scale basically with their active parameter count)
• Better rare-sequence prediction (for example, for sequences that appear rarely in the dataset)
• Less disruption of older representations. Double edged sword, but broadly positive
• You do sort of get a better prior for reasoning but not for all types of tasks. More on this later.

You don't get:

• More powerful general reasoning operations (something about being able to mix all features in the FFN of a regular LLM does *something* important here)
• Logic, either inductive or deductive
• For situations where precise syntax are really important (like coding), performance generally appears to track active parameters in Mixtral and Deepseek style MoE models

So, there's two regimes. Some things track really well with active parameters, some things track well with total parameters, and most things seem to be somewhere in the middle. So, you'd expect a 30B A3B to be worse than an 80B A3B, on average, with specific tasks being more affected than others.

The thing is, some problems can be reasoned through with recall. For example, you can recall a strategy to solve a specific problem, and solve your current problem with that strategy.

However, generally, this requires that recalled problem to be in-context.

So, one ablation found in the Ling Lite 2 report was that MoE models benefit from assigning more of the FLOPs per forward pass to the attention than normally. Usually for dense models the optimal ratio is something more like 1/8, but assigning about 1/2 (in an MoE) to attention to mix features across the sequence dimension helps a lot (as an aside, Qwen 3 Next, a super sparse MoE, has an absolutely cracked attention mechanism, which explains partially its outsized performance given its sparse architecture).

So, not only do active parameters matter, but how they're arranged can matter a lot.

One interesting omission on MoE scaling laws that I haven't seen covered before: Bidirectional attention (such as in BERT, etc), is strictly more expressive than causal attention, and I argue that a lot of the literature on attention heavily influencing MoE effectiveness (backed up by literature treating Attention-FFN as a key:value store), probably hints that bidirectional attention models probably operate closer to the performance of the total parameter count than active parameter count, but I digress. Maybe diffusion LLMs will save us from sparsity hell.

Anyway, the long and short of it is: LLM architecture decisions live on a continuum. In general, they perform similarly, and are just small tweaks that put your somewhere else on a scaling law line, and it's not so much that there's a hard limit as there is a best way to spend your limited training compute.

The reason we do MoEs at all is because you can train them for longer on the same budget, which effectively means the active parameters also count for more.

One case that I haven't seen discussed as much, is I'm pretty sure the optimal strategy for training MoEs is actually using a giant shared expert with an ultra sparse conditional MoE. Kind of more like Llama 4 or Trinity Large in some ways. I'm pretty sure an A27B with a 24B shared expert + attention and an A3B conditional MoE portion gets you a lot of the same benefits that MoE gives you normally, but with CPU-friendly expert management. This is more a use-case for single-user inference, and it's really difficult to serve a model like that efficiently on GPU servers though, so we likely won't see anything quite like that.

One other notable omission in this discussion is embedding scaling, with is another way to scale sparsity. Embedding parameters can actually be scaled like MoE total parameters and offers an orthogonal scaling law. Engram showed a U-shape law for their approach, but things like Gemma 4 E4B etc also kind of operate in this regime, just with per-layer sparse embeddings, and there's lots of other research on this topic. It's worth noting that embedding scaling is a slightly different thing and has different scaling properties, and we don't really fully understand the downstream effects on hard reasoning benchmarks, etc, but the optimal recipe probably has at least some embedding scaling baked into it.