self-attention
In plain wordsAttention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to.
Attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. in which queries, keys and values are all computed from the same sequence, so each position builds a new representation by attending to the positions of that sequence, itself included. From input vectors X a layer forms Q = XW_Q, K = XW_K and V = XW_V with learned matrices and applies scaled dot-product attentionscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information., producing a context-dependent vector for every tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens., computed for all positions in parallel. Self-attention links any two positions in one step, which helps with long-range dependencies, but its time and memory grow quadratically with sequence length n because it forms an n × n score matrix. In decoder-only modelsdecoder-only transformer, n.: The kind of transformer behind most chatbots: a single stack that reads text left to right and continues it, without a separate part that first encodes an input. a causal maskcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat. limits each position to earlier ones. It is the main operation of every transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. layer and is usually split into several headsattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns..
Self-attention cost dominated at 128K tokens, so we switched half the layers to sliding-window attention.
Origin
The transformer paper credits earlier 'intra-attention' within recurrent models for reading and summarization, notably Jianpeng Cheng, Li Dong and Mirella Lapata (2016). Zhouhan Lin and colleagues used the name 'self-attention' for sentence embeddings in March 2017, and 'Attention Is All You Need' (June 2017) made self-attention the sole means of relating positions.
Usage note
Not to be confused with cross-attentioncross-attention, n.: Attention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt., where queries come from one sequence and keys and values from another.
Related
- attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones.
- scaled dot-product attentionscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information.
- multi-head attentionmulti-head attention, n.: Running several attention processes side by side, each free to track a different kind of relationship between words — grammar, topic, who did what — and then combining their results.
- causal maskcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat.
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- query, key and valuequery, key and value, n.: The three roles each word plays inside attention: the query is what it is looking for, the key is what it can be found by, and the value is the information it hands over when chosen.
- cross-attentioncross-attention, n.: Attention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt.
Sources
- Cheng, J., Dong, L. & Lapata, M. (2016). Long Short-Term Memory-Networks for Machine Reading. EMNLP 2016. arxiv.org/abs/1601.06733
- Lin, Z., Feng, M., dos Santos, C. N., Yu, M., Xiang, B., Zhou, B. & Bengio, Y. (2017). A Structured Self-attentive Sentence Embedding. ICLR 2017. arxiv.org/abs/1703.03130
Mentioned in
attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones., context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out., transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., Attention Is All You NeedAttention Is All You Need, prop. n.: The 2017 Google research paper that introduced the transformer, the neural-network design behind ChatGPT, Claude, Gemini and nearly every other modern AI language system., big O notationbig O notation, n.: Shorthand for how the cost of a computation grows as the problem gets bigger. Saying attention costs 'O(n²)' means doubling the length of the text roughly quadruples the work., cross-attentioncross-attention, n.: Attention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt., cross-encodercross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list., decoder-only transformerdecoder-only transformer, n.: The kind of transformer behind most chatbots: a single stack that reads text left to right and continues it, without a separate part that first encodes an input., encoder-only transformerencoder-only transformer, n.: A transformer that reads a whole text at once, in both directions, to understand it rather than to write new text. Such models power search, sorting documents by topic and spotting names in text., long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible., multi-head attentionmulti-head attention, n.: Running several attention processes side by side, each free to track a different kind of relationship between words — grammar, topic, who did what — and then combining their results., positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model., query, key and valuequery, key and value, n.: The three roles each word plays inside attention: the query is what it is looking for, the key is what it can be found by, and the value is the information it hands over when chosen., vision transformervision transformer, n.: A way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence., Ashish VaswaniAshish Vaswani, prop. n.: An Indian-born computer scientist listed first on 'Attention Is All You Need', the 2017 paper that introduced the transformer, the design behind today's language models., ConformerConformer, prop. n.: A neural-network design for speech recognition that combines a transformer, which relates distant parts of a recording, with convolutions, which pick up fine local sound patterns., context parallelismcontext parallelism, n.: A way of training or running a model on extremely long inputs — whole books or codebases — by splitting the text across many chips and having them swap the information each part needs to 'see' the others., hybrid architecturehybrid architecture, n.: An AI model design that mixes two kinds of building blocks — usually a few standard attention layers among cheaper, memory-saving layers — to keep most of the quality at lower cost on long inputs., Jakob UszkoreitJakob Uszkoreit, prop. n.: A German computer scientist, one of the eight authors of the transformer paper, credited with first proposing to build language models on attention alone., rotary positional embeddingrotary positional embedding, n.: A way of telling a transformer where each word is by rotating parts of its internal numbers by an angle that grows with position, so the model can read off how far apart two words are., sparse attentionsparse attention, n.: Attention in which each word looks at only a selected subset of the other words instead of all of them, so very long texts can be processed at a fraction of the cost.