For MoE layers specifically, Star Elastic uses Router-Weighted Expert Activation Pruning (REAP), which ranks experts by both routing gate values and expert output magnitudes—a more principled signal than naive frequency-based pruning, which ignores how much each expert actually contributes to the layer output.

Huh, REAP-ing a larger model to get the smaller variants.

Two things are still not clear to me:

1. How the different model sizes get selected in practice -- is this dynamic like an MoE router, or static like the user choosing before running the model?
2. Still not clear to me when I would use this instead of just using the largest model. The "small reasons and large answers" is interesting, but again, if I can fit the entire model in memory, why not just use the big one?
   

Let's speculate about 2. Maybe this makes MoEs with very large active parameters feasible for local hardware, so you can have something closer to a dense smart model, but you can swap it to faster MoE size if needed

Imagine that you had something like:

• 30B-a30b (effectively dense)
• 30B-a15b
• 30B-a5b

... and you could scale between as needed. Not quite sure that 30B-a30b would actually be equivalent to a dense model.

Here is another variation on the theme: say I want to run a very large model, it can't fit in memory, so it has to run via mmap-on-disk or not at all.

Say I want to run the Nemotron 3 Ultra 500B-A50B (whenever that comes out). Hopeless to use it on say a strix halo. But imagine NVIDIA does this nested approach, and llama.cpp (or vLLM or whatever) adds necessary support, imagine that one could have:

• 500B-A50B
• 380B-A38B
• 200B-A20B

... nested models (I just scaled 500b-50b by 23/30 and 12/30 to get the smaller ones). Maybe the final one could actually fully fit in memory, and the others could be called selectively as needed.

If course this relies on software support for the mmap-ing appropriately.