attention sink
In plain wordsA quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need.
A token, usually the first in the sequence or a beginning-of-sequence marker, that receives a disproportionately large share of attention weight across many heads and layers regardless of its content. Because softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score. weights must sum to one, a head that has nothing useful to look at must still place its weight somewhere, and models learn to send that surplus to a position that is always visible and whose value vector adds little. Sinks matter in practice: evicting the first tokens from a sliding-windowsliding window attention, n.: Attention in which each word looks only at a fixed number of nearby earlier words rather than the whole text, keeping the cost of very long inputs under control. KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. makes perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll. explode, while keeping a few sink tokens, as in StreamingLLM, restores stable generation over millions of tokens. They are associated with very large activations that complicate quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.. Remedies include a dedicated learned sink token, an extra constant in the softmax denominator, or gating of the attention output.
Keep the four sink tokens when you evict old cache entries, or quality collapses once the window fills.
Origin
Named by Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han and Mike Lewis in 'Efficient Streaming Language Models with Attention Sinks' (arXiv, September 2023; ICLR 2024). Heavy attention to special tokens had already been noticed in BERT by Kevin Clark and colleagues (2019).
Usage note
Evan Miller's 2023 post 'Attention Is Off By One' proposed adding 1 to the softmax denominator for a related reason. Some 2025 open-weight models, such as OpenAI's gpt-oss, build learned sink terms into attention.
Related
- attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones.
- softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.
- sliding window attentionsliding window attention, n.: Attention in which each word looks only at a fixed number of nearby earlier words rather than the whole text, keeping the cost of very long inputs under control.
- KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.
- long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
Sources
- Xiao, G., Tian, Y., Chen, B., Han, S. & Lewis, M. (2023). Efficient Streaming Language Models with Attention Sinks. arXiv:2309.17453; ICLR 2024. arxiv.org/abs/2309.17453
- Clark, K., Khandelwal, U., Levy, O. & Manning, C. D. (2019). What Does BERT Look At? An Analysis of BERT's Attention. BlackboxNLP 2019.
Mentioned in
lost in the middlelost in the middle, phr.: AI models reading a long input behave a little like a skimmer: they reliably catch facts placed near its start or end but are much likelier to miss the same fact buried halfway through., streamingstreaming, n.: Sending an AI's answer piece by piece while it is still being written, so the words appear on screen one after another instead of all at once at the end., attention patternattention pattern, n.: A grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map., KV cache compressionKV cache compression, n.: Techniques that shrink the memory an AI model uses to remember a long conversation or document, by storing its notes more compactly or discarding the least useful ones., sliding window attentionsliding window attention, n.: Attention in which each word looks only at a fixed number of nearby earlier words rather than the whole text, keeping the cost of very long inputs under control.