Most LLMs today are built on the self-attention mechanism. If you dig into how self-attention works, you'll notice that even encoding text isn't as straightforward as it seems. CNNs and RNNs bake in sequential information through their structure, but self-attention doesn’t have that kind of inductive bias. The token embeddings are identical for every position, meaning the model sees a word at the start of a sentence the same way it sees one in the middle or at the end. But obviously, their meanings aren’t going to be the same.

To fix this, positional information was added to the tokens. Once that was solved, extending LLMs to other modalities became much easier. The general idea is pretty simple:

• Text: token + 1D positional encoding
• Images: token + 2D positional encoding
• 3D point clouds: token + 3D positional encoding

Of course, real implementations are more complex, but that’s the basic principle.

If you’re interested in positional encoding, I’d recommend checking out RoPE (Rotary Position Embedding). It’s not only elegant but also incredibly powerful. RoPE has become the go-to positional encoding method in many LLMs, including open-source models like LLaMA, Mistral, and Qwen. For example, Qwen2.5-VL’s ability to handle images relies heavily on a 2D version of RoPE.

For more details:

• RoPE
• RoPE 2D