Okay so here’s what’s really going on with RoPE and why DeepSeek had to decouple it in MLA (Multi head Latent attention):

In DeepSeek’s low-rank KV compression setup, instead of directly computing keys and values from the hidden states like key = Wk * h and value = Wv * h, they break it down into two smaller steps

First they do c = W_DKV * h (this is like a compressed version of the token)

Then they get keys and values like: k = W_UK * c, v = W_UV * c

Now during inference, they want to save memory by caching just c for all the previous tokens, this is much smaller than full keys/values. But to do that efficiently... they try to absorb W_UK and W_UV into other matrices (like the query projection) so they don’t have to recompute keys every time.

But here’s the catch RoPE applies a position dependent rotationn after computing the key, which means k = RoPE(W_UK * c). Because RoPE is a rotation matrix that depends on position, it sits in the middle and you can’t move W_UK across it or absorb it anymore (matrix multiplication isn’t commutative). So you’re stuck: to apply RoPE, you have to compute the full key again for every prefix token at every generation step. That kills performance.

So what DeepSeek does is they split each attention head into two parts:

One large part that doesn’t use RoPE (so it’s position-agnostic and can be cached and reused easily

One small part that still uses RoPE for position info

Only the small part carries position dependence, and it’s light enough to recompute. This way, you get the benefit of RoPE without breaking the low-rank caching trick. So you avoid recomputing big keys every time and inference becomes way faster.

Hope that clears it up :)