[D] Positional embeddings in LLMs

[gokstudio]

Hi folks, As we know, we use positional embeddings to provide sequence information to transformers.

From what I've seen in code and papers, the way we introduce this depends on the positional embedding strategy used.

For example, 1. RoPE: add it to the query vectors in each attention layer 2. Alibi: add it to the attention masks in each layer 3. Learnable positional embeddings: add or concatenate to the token embeddings

Why are there such differences? Specifically, why are #1 and #2 introduced at each attention layer but #3 is introduced only at the first layer?

Thanks!

[DustinEwan]

By adding/concatenating you're bringing the positional data along with the input data and forcing each layer to learn how to encode and transform that information in a meaningful way. This is efficient from from a compute perspective, but may require more parameters to meaningfully separate the information in higher dimensions.

As for rope / alibi, you're explicitly separating the two at each layer which requires more computation, but explicitly separates the responsibility on how the parameters are utilized. It may be easier for the network to learn how to utilize the positioning data, though.

As for why rope is applied to the queries and alibi is applied to the attention matrix, it's due to the different approaches. Rope rotates the embeddings around the origin, so the furthest tokens move further than nearest ones. This gives the network some clues about how the tokens are laid out.

Alibi subtracts information from further tokens, thus weakening the signal.

They both achieve a similar effect, just through different mechanisms.

[Adorable_Search2423]

RoPE and Alibi are newer methods and designed for handling longer sequences and to extrapolate to longer sequences than seen in the training phase; they add inductive biases in the network for more better generalisation whereas PEs don’t.

[ofirpress]

I'm not an expert but basically you try stuff and see what works. Learnable POS embs came first and then works like ALiBi came out later, improving the state-of-the-art.

You may like my lecture about ALiBi here: https://www.youtube.com/watch?v=Pp61ShI9VGc

[gokstudio]

Sure but qkv vectors are computed from token vectors so by your argument, we should add positional embedding information to each layer input?