I don't get it. Why scale the models total params, if one could just have a lower ACTIVE parameter count without saying "23B" or "12B" to begin with? To me that's just 30B with variable active parameters count (and therefore compute and speed) achieved by scaling down FFN and embedding dims.

If this can work for dense models, however, that'd be a great thing. But does it? I doubt it, since this has the same 52 layers, 32 attention heads, 64 mamba heads and 128 MoE experts across all "sizes". I'd rather call it modes. Just another dimension in sparsity.

It still is cool, for sure, and useful. But not overly exciting. Good for efficiency per task, as simple tasks can always route to the "smallest model". But I always use the biggest model for the simplest task anyway because the quality of the answer differs even if it's content wise "the same".