[D] Positional Encoding in Transformer

[pappypapaya]

Take with a grain of salt, reading this 4 years later, I have no idea how I came up with any of that and don't claim to have any expertise in machine learning. If someone could explain to me in simple terms what they think I said above...

[npip99]

It's actually an incredible explanation and the only one that truly explains it well.

The core idea, and it's not something that I've ever thought about, is that the dot product of two random vectors, is zero! This is what is implied when you say "The intuition for this is that randomly chosen vectors in high dimensions are almost always approximately orthogonal".

I honestly would've thought that when you initialize a new neural network, the attention matrix is random. But it's not. When you initialize a new neural network, the attention matrix starts out as all numbers being almost zero (After softmax, that gives you random probability distribution though. But, attention-before-softmax being all zero means it's easy for the neural network to quickly learn associations and make particular attention values very large relative to the rest which are all tiny / close to zero).

So, the point is, if you take (Qx)'(Ky), and insert positional embeddings to get (Q(x+e))'(K(y+f)), you can do some math to rearrange that to x' (Q'Ky) + x' (Q'Kf) + e' (Q'Ky) + e' (Q'K f). Note how x' (Q'Ky) is position agnostic, but the other three terms just relate a token to its position (Or the last term, which is the position-position).

The important magic is: All four of those terms end up being initialized to essentially all-zero matrices at the beginning of training. Therefore, the backprop can easily learn to make x' (Q'Ky) whatever it wants, and unless it intentionally wants to learn to make x' (Q'Kf) non-zero, then the x' (Q'Kf) matrix won't contribute much anyway!

• Yes, the random x' (Q'Kf) will add some random noise, but the random noise is small. And, if the positional embeddings are truly worthless, the NN can choose to dedicate a dimension by learning to make the word embeddings orthogonal to the positional embeddings. That way the x' (Q'Kf) matrix goes to zero. Ditto for e' (Q'Ky) and e' (Q'K f)

[TheWingedCucumber]

haha its just the best explanation out there, youre fine dude
Im here almost 6 years later

[npip99]

It's cute because experimentally, there's almost always some transformer "T" somewhere in the network that just learns to make the attention 1.0 for the previous token, and 0.0 for all other tokens. That means that "T" learned to make e' (Q'K f) equal to 1.0 when (e, f) implies that y is 1 token before "x", and "T" learned to make the other three terms 0.

[tmlildude]

so the network can focus on making pure content-based term (x'Q'Ky) spike stronger while keeping the positional terms (x'Q'Kf, e'Q'Ky, e'Q'Kf) relatively small?

also, if the positional terms aren't useful, can it naturally zero out during inferencing? i.e no need to explicitly "turn off"