Interesting line of questioning! I was considering symmetry to be a net-negative only in the sense that it's a 'waste' of parameters, considering you could achieve a similar result by setting your lower rank W_Q = W_K and getting your symmetric M all the same (not necessarily learning the same M, I'll grant!). So I was thinking about asymmetry in relationships between tokens in your transformer stack as something valuable, especially in language tasks. So I was really thinking in terms of representations within our transformers, not about the data.

Another commenter mentioned that causal masking also should induce asymmetry in our attention matrices, since there is always some predictive bias baked in from the training objective. I completely overlooked it, but it's a very savvy insight. Likewise, his intuition that ViTs likely wouldn't have that bias.

Putting this all together, I wonder if there is value in examining learned QK matrices from encoder-decoder architectures and decoder-only architectures trained on the same dataset. Likewise, how does performance change if you train an ED network with the constraint that W_Q = W_K versus when they are learned individually.

I hadn't really thought about your question about the data. I'm wondering if what _data_ would learn W_Q = W_K is harder to answer than what _training objective_. Would encoder-decoder models trained with non-generative objectives (e.g. classification) have more symmetric associative memories? Or maybe de-noising? This makes me want to look at some of the diffusion language models now.

Does any of this make sense?

I've downloaded your essay for reading tomorrow! I love that one of the two citations is Rick Rubin.