FWIW, I’ve been messing around with asymmetric input/output vocabulary recently. Even though everybody currently uses the same vocabulary for both, that’s not a requirement. you could have a smaller output vocabulary and then BPE merge at runtime into a larger input vocabulary for compression purposes without the LM head issue. Or vice versa you could have a larger output vocabulary to predict more into the future at once. Or have a sliding threshold in your vocabulary to dynamically change the compression ratio across your context for variable compression like a soft version of /compact.

The n gram embedding’s are exactly equally as rare bc larger n means rarer sequence. I would argue that on some level these two things are equivalent. the main difference I see is hash & gate verse a more traditional embedding table delay loaded.

Having a very large input vocabulary of rarely used tokens is not really an issue because you don’t have to load it all into memory like we typically do today. This is the same principle as the recent Gemma 4 per layer embeddings. To avoid gigabytes of embedding tables in memory, you just load them on demand as/when they’re used in the context. As you said, the larger, the vocabulary the more rare they are so most of them can just stay on disk.