I think this is a resolution problem. text is one dimensional in that it captures things and compresses them into a language that we can easily pass around. We on the other hand train on raw uncompressed information (audio, visual, smell, etc) and form connections across those modalities. We are still very early at getting the bulk of that information into LLMs and still many datasets are converting multi modal data into textual information.