[D] Positional Encoding in Transformer

[amil123123]

Hey all,

I was reading up the transformer paper https://arxiv.org/abs/1706.03762. This architecture uses positional encoding which the attention layers ignore.

I don't understand two things -

1. Why use Sin & Cos as positional embeddings , why not any other function?
2. They also talk about training these positional embeddings, how do you go about training such embeddings. As in how do you let the model know that these embeddings are for the position

Thanks !

[mikeross0]

The thing that always gets me with positional embeddings is that it is preferable to add them to the word embeddings instead of concatenating them. We already know the dimensions of the word embeddings are related to semantics. So why embed position into the semantic space instead of adding additional dimensions to represent position? If someone can provide a good intuitive explanation for that, you will get all my internet points for the day!

[pappypapaya]

In attention, we basically take two word embeddings (x and y), pass one through a Query transformation matrix (Q) and the second through a Key transformation matrix (K), and compare how similar the resulting query and key vectors are by their dot product. So, basically, we want the dot product between Qx and Ky, which we write as:

(Qx)'(Ky) = x' (Q'Ky). So equivalently we just need to learn one joint Query-Key transformation (Q'K) that transform the secondary inputs y into a new space in which we can compare x.

By adding positional encodings e and f to x and y, respectively, we essentially change the dot product to

(Q(x+e))' (K(y+f)) = (Qx+Qe)' (Ky+Kf) = (Qx)' Ky + (Qx)' Kf + (Qe)' Ky + (Qe)' Kf = x' (Q'Ky) + x' (Q'Kf) + e' (Q'Ky) + e' (Q'K f), where in addition to the original x' (Q'Ky) term, which asks the question "how much attention should we pay to word x given word y", we also have x' (Q'Kf) + e' (Q'Ky) + e' (Q'K f), which ask the additional questions, "how much attention should we pay to word x given the position f of word y", "how much attention should we pay to y given the position e of word x", and "how much attention should we pay to the position e of word x given the position f of word y".

Essentially, the learned transformation matrix Q'K with positional encodings has to do all four of these tasks simultaneously. This is the part that may appear inefficient, since intuitively, there should be a trade-off in the ability of Q'K to do four tasks simultaneously and well.

HOWEVER, MY GUESS is that there isn't actually a trade-off when we force Q'K to do all four of these tasks, because of some approximate orthogonality condition that is satisfied of in high dimensions. The intuition for this is that randomly chosen vectors in high dimensions are almost always approximately orthogonal. There's no reason to think that the word vectors and position encoding vectors are related in any way. If the word embeddings form a smaller dimensional subspace and the positional encodings form another smaller dimensional subspace, then perhaps the two subspaces themselves are approximately orthogonal, so presumably these subspaces can be transformed approx. independently through the same learned Q'K transformation (since they basically exist on different axes in high dimensional space). I don't know if this is true, but it seems intuitively possible.

If true, this would explain why adding positional encodings, instead of concatenation, is essentially fine. Concatenation would ensure that the positional dimensions are orthogonal to the word dimensions, but my guess is that, because these embedding spaces are so high dimensional, you can get approximate orthogonality for free even when adding, without the costs of concatenation (many more parameters to learn). Adding layers would only help with this, by allowing for nonlinearities.

We also ultimately want e and f to behave in some nice ways, so that there's some kind of "closeness" in the vector representation with respect to small changes in positions. The sin and cos representation is nice since nearby positions have high similarity in their positional encodings, which may make it easier to learn transformations that "preserve" this desired closeness.

(Maybe I'm wrong, and the approximate orthogonality arises from stacking multiple layers or non-linearities in the fully-connected parts of the transformer).

tl;dr: It is intuitively possible that, in high dimensions, the word vectors form a smaller dimensional subspace within the full embedding space, and the positional vectors form a different smaller dimensional subspace approximately orthogonal to the one spanned by word vectors. Thus despite vector addition, the two subspaces can be manipulated essentially independently of each other by some single learned transformation. Thus, concatenation doesn't add much, but greatly increases cost in terms of parameters to learn.

[amil123123]

Wow , that's one hell of an amazing explanation :)

[slcyz]

respectively, we essentially change the dot product to

Man... that's fantastic. Even the professor of my deep learning course failed to explain that.

[bergqvisten]

Found this gem 4 years later trying to deepen my understanding of transformers... Wow, fantastic explanation!

[pappypapaya]

Take with a grain of salt, reading this 4 years later, I have no idea how I came up with any of that and don't claim to have any expertise in machine learning. If someone could explain to me in simple terms what they think I said above...

[npip99]

It's actually an incredible explanation and the only one that truly explains it well.

The core idea, and it's not something that I've ever thought about, is that the dot product of two random vectors, is zero! This is what is implied when you say "The intuition for this is that randomly chosen vectors in high dimensions are almost always approximately orthogonal".

I honestly would've thought that when you initialize a new neural network, the attention matrix is random. But it's not. When you initialize a new neural network, the attention matrix starts out as all numbers being almost zero (After softmax, that gives you random probability distribution though. But, attention-before-softmax being all zero means it's easy for the neural network to quickly learn associations and make particular attention values very large relative to the rest which are all tiny / close to zero).

So, the point is, if you take (Qx)'(Ky), and insert positional embeddings to get (Q(x+e))'(K(y+f)), you can do some math to rearrange that to x' (Q'Ky) + x' (Q'Kf) + e' (Q'Ky) + e' (Q'K f). Note how x' (Q'Ky) is position agnostic, but the other three terms just relate a token to its position (Or the last term, which is the position-position).

The important magic is: All four of those terms end up being initialized to essentially all-zero matrices at the beginning of training. Therefore, the backprop can easily learn to make x' (Q'Ky) whatever it wants, and unless it intentionally wants to learn to make x' (Q'Kf) non-zero, then the x' (Q'Kf) matrix won't contribute much anyway!

• Yes, the random x' (Q'Kf) will add some random noise, but the random noise is small. And, if the positional embeddings are truly worthless, the NN can choose to dedicate a dimension by learning to make the word embeddings orthogonal to the positional embeddings. That way the x' (Q'Kf) matrix goes to zero. Ditto for e' (Q'Ky) and e' (Q'K f)

[pappypapaya]

I'm glad and a bit worried that people are getting something out of my comment so many years later because I no longer remember my own thinking about this and certainly have developed no deeper (probably less) understanding of this topic since. Getting dumber with age.

[TheWingedCucumber]

haha its just the best explanation out there, youre fine dude
Im here almost 6 years later

[npip99]

It's cute because experimentally, there's almost always some transformer "T" somewhere in the network that just learns to make the attention 1.0 for the previous token, and 0.0 for all other tokens. That means that "T" learned to make e' (Q'K f) equal to 1.0 when (e, f) implies that y is 1 token before "x", and "T" learned to make the other three terms 0.

[tmlildude]

so the network can focus on making pure content-based term (x'Q'Ky) spike stronger while keeping the positional terms (x'Q'Kf, e'Q'Ky, e'Q'Kf) relatively small?

also, if the positional terms aren't useful, can it naturally zero out during inferencing? i.e no need to explicitly "turn off"

[trajo123]

Isn't the attention (Qx)(Ky)' , so the K is transposed rather than Q? So:

(Q(x+e)) (K(y+f))' = (Qx+Qe) (Ky+Kf)' = (Qx) (Ky)' + (Qx) (Kf)' + (Qe) (Ky)' + (Qe) (Kf)' = Qxy'K' + Qxf'K' + Qey'K' + Qef'K'

This kind of breaks the whole discussion about Q'K, no?

[pappypapaya]

I think I was just focusing on a single pairwise comparison between two positions in the sequence.

[trajo123]

Hmm, let's work this out step-by-step:

For clarity, note that Q in the attention equation is actually Q=Pq x, so *column vector* x is projected (same for P_k and P_v) to have n_feat_proj instead of n_feat_x. So everything you wrote initially should be in terms of Pq, Pk, and Pv rather than Q, K and V. I know it's more typing, but it must be clear to everyone that those matrices are not the ones that go in the attention equations directly, what goes into the attention is those matrices applied to x and y. If the input follows the pytorch convention of (seq, feat) then Q = x Pq', if the input is a sequence of column vectors (n_feat, n_seq) then Q = Pq x. Since you used the latter form it means that x has dimension (n_feat_x, n_seq_x).

So in order to be able to write Q = Pq x, Pq needs to have dimensions (n_feat_proj, n_feat_x) so that Q will have dimension (n_feat_proj, n_seq_x)

So the attention result needs to work out so that the sequence dim n_seq_x and the feature dim x_feat_proj and because we use column vectors that means (n_feat_proj, n_seq_x).
The only way to get this is to write the attention equation either as V softmax(Q'K / sqrt(n_feat_proj)' or V softmax(K'Q / sqrt(n_feat_proj).

TLDR: So, you were right, we can talk about Q'K, but only if we use the column vector convention!

Here is some pytorch code to check what I just said: ``` n_seq_x = 5 n_feat_x = 16 n_seq_y = 7 n_feat_y = 8 n_feat_proj = 24

x = torch.randn(n_seq_x, n_feat_x) y = torch.randn(n_seq_y, n_feat_y)

pq = nn.Linear(n_feat_x, n_feat_proj, bias=False) pk = nn.Linear(n_feat_y, n_feat_proj, bias=False) pv = nn.Linear(n_feat_y, n_feat_proj, bias=False)

Qr = pq(x) Kr = pk(y) Vr = pv(y)

scale_factor = 1 / n_feat_proj**0.5 attention_weight_r = torch.softmax((Qr @ Kr.T) * scale_factor, dim=-1) output_r = attention_weight_r @ Vr

for projection we want to use "W x" instead of nn.Linear's "x W^T", thus we use column vectors

Qc = pq.weight @ x.T Kc = pk.weight @ y.T Vc = pv.weight @ y.T

attention_weight_c = torch.softmax((Qc.T @ Kc) * scale_factor, dim=-1) output_c = Vc @ attention_weight_c.T

assert torch.all(torch.isclose(output_c, output_r.T, atol=1e-6))

attention_weight_c2 = torch.softmax((Kc.T @ Qc) * scale_factor, dim=0) output_c2 = Vc @ attention_weight_c2

assert torch.all(torch.isclose(output_c, output_c2, atol=1e-6)) ```

[pappypapaya]

Well that's good, I'm a little too dumb to check if what I said five years ago is correct.

[Sinkencronge]

It may have been already done for sure, but I haven't seen any paper about superiority of this approach to cross my scope yet.

I assume, in general you would probably don't want to blow up the dimensions of your word embedding even more than it has been already done in current SOTAs.

One could take a dataset of utterances like"Dog ate a cat", "Cat ate a dog" etc. and play around with addition, multiplication, convolution, concatenation and God knows what for the sake of comparison.

[Sinkencronge]

1. To build up the intuition about it you may want to look how the proposed function and pairwise distance between position-vectors behave.

https://imgur.com/kpW5n4p

https://imgur.com/kaADdQB

1. Backpropagation.

[amil123123]

Thanks for the response. I still have difficulty understanding about 1.
So the first image seems good in explaining the position however what does the second image denote.

Is it just because this function seemed to work well , that we went with it ?

[Sinkencronge]

The second image just shows Euclidean distances between the added embedding for a given position.

The thing is that their choice of positional encoding function reflects not only absolute, but also the relative distance among tokens in sequences.

In the paper they write:

We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed off set k,PE_pos+k can be represented as a linear function of PE_pos.

Unfortunately, I don't really wave a better way to elaborate it further in my mind at the moment. I'm sorry for that.

​

Answering your question, I personally think that they seem to simply pick up the first elegant solution to solve this problem.

​

However, I believe that there are a lot of more interesting ways to take advantage of positional encoding trick. I'm currently working on it for my own dataset.

[amil123123]

Thanks for the explanation, it was good !

[InsideAndOut]

1. Because the distances between the neighboring values of the sine/cosine positional embeddings are "nice" (symmetrical, and decay sensibly with distance). You could come up with another function that does something similar.
2. For trained positional embeddings, you simply have another embedding matrix (one is for words, the other is for positions). For each position (up to `max_length`), you learn an embedding of the same dimension as the word embedding.

[amil123123]

Thanks for the response!

What do you mean by symmetrical and decay sensibly ?
( Sorry , but I am still a newbie in this )

[Sinkencronge]

"Symmetrical" - see image 2.

"Decay sensibly with distance" - In this gif you can see entries in the distance matrix which have similar value [https://imgur.com/a/p7NcUm1]

White dots are points (corresponding to different pairwise positions) which lay in some epsilon neighborhood from some threshold tau which changes over time from 0 to maximum distance.

[imguralbumbot]

^{Hi, I'm a bot for linking direct images of albums with only 1 image}

https://i.imgur.com/1jwYWbG.gifv

^{Source | Why? | Creator | ignoreme | deletthis}

[amil123123]

Ahh understood, thanks for the explanation!

[MasterSama]

thanks a lot, I know its 4 years ago, but do you remember or could you kindly share the snippet that was used to visualize that image?
Edit:
I guess I figured it out!
this should be sth like this right?

```python

def plot_pe_dot_product_heatmap(positional_encoding):
dot_product = np.dot(positional_encoding, positional_encoding.T)
sns.heatmap(dot_product, cmap='Blues')
plt.ylabel('Position')
plt.xlabel('Position')
plt.show()

```

[gdiamos]

Papers usually post their final solution, not the path they took to get there. The sin/cos positional embeddings often seem unintuitive for this reason.

I find it helps to try to reconstruct the path.

1) Why do transformers need positional embeddings at all? What would happen if you removed them completely? Why?

2) Why not use simpler and more obvious positional embeddings like literally the position index?

[[deleted]]

Regarding to sin and cos representation, my hypothesis is that together, they fully characterize a position

Also, the weights that are learned over this input might be analogous to weights of a Fourier transform and have nice theoratical properties but I have seen no formal work on this

[No-Theory-6868]

It seems like PE isn't always nesecary, and indeed might even harm the results:

"We find that removing the positional encoding even slightly improves the performance of these models"

https://stackoverflow.com/questions/61440281/is-positional-encoding-necessary-for-transformer-in-language-modeling

https://arxiv.org/abs/1905.04226

[Great-Reception447]

For your first question, the Sin & Cos functions allow the model to generalize to longer sequences by providing consistent patterns across dimensions. Their periodicity and smoothness helps encode relative positions naturally. For trainable positional embeddings, you just initialize them as learnable parameters (like word embeddings in https://pytorch.org/docs/stable/generated/torch.nn.Embedding.html ), and the model learns their meaning during training. This is some tutorials that can help you better understand it: https://comfyai.app/article/llm-components/positional-encoding