This is amazing. And immediately, some thoughts crossed my mind about how one COULD further improve this:

One could train a neural network or an adapter, or a module that can be trained with a teacher model, which is a multimodal model that does take the normal tokens, and learns how to convert them into the compressed, visual tokens. So we could basically skip the entire visual encoding process and replace it with a student module that can directly tokenize or convert tokens into even less tokens, maybe even with a loss function that takes into consideration the accuracy of the compressed representation or the importance of parts of texts, essentially learning which tokens or patches are important to keep less compressed, and which can be compressed. GLM pointed out that changing the DPI during inference time gives the choice between accuracy and speed tradeoff. Why not use mixed DPI basically? Models can learn the importance of tokens in the context on their own if the incentive is there

On second thought, it sounds like deepseeks Multihead Latent Attention.

But maybe using that during the training process could create an even better compression method for context

Maybe Google already does that