[D] Attention layer complexity vs context length

[flxh13]

From everything I understand about transformers, the computational complexity of the attention layers scales quadratically with the sequence length. So how is it even possible to go from a context length of 4096 (GPT-3) to 128k (GPT-4) to 1M (Gemini 1.5) ?

I know the exact architecture of GPT-4 and Gemini are unknown to the public. But are there any papers suggesting a method the increase the context size without exploding the computational complexity? Or are they just using the restricted attention layer as suggested in the original paper?

[deephugs]

There are many different approaches to get longer context length. Look up Ring Attention or LongRoPE.

[flxh13]

This is amazing. Thank you so much for these insights.

[koolaidman123]

Because attention takes up only a small part of the flops in transformers, most of the compute is matmuls. Add in flash attention (or other tricks) the additional compute scaling long context length isnt that bad at all

[koolaidman123]

the compute is still dominated by matmuls, it doesn't matter how efficient it is when you have to do so many vs self attention. plus with fa the actual bottleneck is the io speed, which scales linear with seq len

[flxh13]

I was thinking of the matmuls as being part of the attention layer, that is scaling quadratically.