sliding window attention
In plain wordsAttention in which each word looks only at a fixed number of nearby earlier words rather than the whole text, keeping the cost of very long inputs under control.
A form of sparse attentionsparse attention, n.: Attention in which each word looks at only a selected subset of the other words instead of all of them, so very long texts can be processed at a fraction of the cost. in which each position attends only to the previous w positions (a band-shaped attention maskattention mask, n.: A grid of yes-or-no switches telling an attention layer which words each word may look at — for example, ignore blank padding, or do not peek at words that come later.), so cost per token is constant, total cost grows linearly with sequence length, and that layer's KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. never exceeds w entries. Information can still travel farther than w through stacking: after k layers a position can be influenced by tokens roughly k × w back. Pure sliding windows struggle with precise retrieval from far away, so models usually interleave local layers with global full-attention layers — GPT-3 alternated dense and banded layers, Gemma 2 alternates local and global layers, and Gemma 3 uses five local layers per global one — or add a few global tokens, as in Longformer. Evicting the first tokens from a window can destabilize models, which led to retaining attention sinksattention sink, n.: A quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need..
Five of every six layers use a 1,024-token sliding window, which keeps the cache small at long context.
Origin
Local attention windows appear in the Image Transformer (2018) and the Sparse Transformer (2019). Longformer (Iz Beltagy, Matthew E. Peters and Arman Cohan, April 2020) combined sliding windows with task-specific global attention, and Mistral 7B (2023) popularized SWA in open LLMs.
Related
- sparse attentionsparse attention, n.: Attention in which each word looks at only a selected subset of the other words instead of all of them, so very long texts can be processed at a fraction of the cost.
- attention maskattention mask, n.: A grid of yes-or-no switches telling an attention layer which words each word may look at — for example, ignore blank padding, or do not peek at words that come later.
- long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.
- KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.
- attention sinkattention sink, n.: A quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need.
- linear attentionlinear attention, n.: A family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words.
Sources
- Beltagy, I., Peters, M. E. & Cohan, A. (2020). Longformer: The Long-Document Transformer. arXiv:2004.05150. arxiv.org/abs/2004.05150
Mentioned in
attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones., attention maskattention mask, n.: A grid of yes-or-no switches telling an attention layer which words each word may look at — for example, ignore blank padding, or do not peek at words that come later., big O notationbig O notation, n.: Shorthand for how the cost of a computation grows as the problem gets bigger. Saying attention costs 'O(n²)' means doubling the length of the text roughly quadruples the work., long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible., attention sinkattention sink, n.: A quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need., hybrid architecturehybrid architecture, n.: An AI model design that mixes two kinds of building blocks — usually a few standard attention layers among cheaper, memory-saving layers — to keep most of the quality at lower cost on long inputs., sparse attentionsparse attention, n.: Attention in which each word looks at only a selected subset of the other words instead of all of them, so very long texts can be processed at a fraction of the cost.