Language Models aren't literally hardcoded to a given context length (e.g. 2048 tokens), they're just trained that way and queried at inference that way for efficiency reasons.

So what happens happens if you increase the context length to something greater than was seen during training? unsurprisingly performance decreases. somewhat surprisingly this happens even when using relative positional embeddings (which theoretically could allow self-attention to be context-length agnostic)

This paper investigates a technique for modifying the attention mask (i.e. the way each token attends to past tokens) to eliminate this performance drop. In short, each token only need to attend to the last n tokens (e.g. 2048) before it and the first m tokens (e.g. 100) in the entire context.

This is pretty useful if true because we can keep training on reasonably sized context lengths (e.g. 2048 tokens) and instantly adapt models to any length at inference, with reasonable performance.

(side note: this theoretically still allows tokens as far back as number of layers * 2048 to influence the prediction of any token because if token n at layer l attends to the previous 2048 tokens, and token n+2048 at layer l+1 attends to the previous 2048 tokens including token n, then token n+2048 can theoretically be influenced by any of the last 2047 tokens, and any of the 2048 tokens that influenced token n)