attention head
In plain wordsOne of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.
A single attention computation within a multi-head attentionmulti-head attention, n.: Running several attention processes side by side, each free to track a different kind of relationship between words — grammar, topic, who did what — and then combining their results. layer, with its own learned query, key, value and output projections into a low-dimensional subspace, typically 64–128 dimensions. Each head produces its own attention patternattention pattern, n.: A grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map. over positions and writes its result back into the residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document. through its slice of the output projection. Interpretability research has found heads with recognizable jobs — attending to the previous token, copying earlier text through induction headsinduction head, n.: A pattern-completing mechanism inside language models: having seen 'Harry Potter' once, it spots the next 'Harry' and predicts 'Potter' — a simple trick behind much learning from examples in a prompt., tracking syntactic relations — while many others are redundant: 2019 studies showed that a large share of heads can be pruned with little loss. Architectural variants change what heads share: in multi-query attentionmulti-query attention, n.: A cheaper form of attention in which all the attention heads share one store of information about earlier words instead of each keeping its own, making text generation faster and lighter on memory. and grouped-query attentiongrouped-query attention, n.: A middle ground between full multi-head attention and its cheapest variant: attention heads are split into groups, and each group shares one store of information about earlier words, saving memory with little loss in quality., several query heads share one key–value head.
Ablating head 5.1 broke the model's ability to close brackets.
Origin
Introduced with multi-head attention in 'Attention Is All You Need' (Vaswani et al., 2017). Paul Michel, Omer Levy and Graham Neubig ('Are Sixteen Heads Really Better than One?') and Elena Voita and colleagues showed in 2019 that many heads can be removed; Anthropic's circuits work from 2021 characterized individual heads mechanistically.
Related
- multi-head attentionmulti-head attention, n.: Running several attention processes side by side, each free to track a different kind of relationship between words — grammar, topic, who did what — and then combining their results.
- attention patternattention pattern, n.: A grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map.
- induction headinduction head, n.: A pattern-completing mechanism inside language models: having seen 'Harry Potter' once, it spots the next 'Harry' and predicts 'Potter' — a simple trick behind much learning from examples in a prompt.
- attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones.
- residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.
- QK and OV circuitsQK and OV circuits, n.: A way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees.
Sources
- Michel, P., Levy, O. & Neubig, G. (2019). Are Sixteen Heads Really Better than One? NeurIPS 2019.
- Voita, E., Talbot, D., Moiseev, F., Sennrich, R. & Titov, I. (2019). Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Pruned. ACL 2019.
Mentioned in
attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones., ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network., activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model., activation vectoractivation vector, n.: The list of numbers a neural network produces at one layer for one input — a snapshot of its internal state. In steering, also a direction added to that state to shift behavior., circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name., induction headinduction head, n.: A pattern-completing mechanism inside language models: having seen 'Harry Potter' once, it spots the next 'Harry' and predicts 'Potter' — a simple trick behind much learning from examples in a prompt., multi-head attentionmulti-head attention, n.: Running several attention processes side by side, each free to track a different kind of relationship between words — grammar, topic, who did what — and then combining their results., pruningpruning, n.: Cutting away the parts of an AI model that contribute least, as a gardener trims branches, so the model becomes smaller or faster while still working nearly as well., query, key and valuequery, key and value, n.: The three roles each word plays inside attention: the query is what it is looking for, the key is what it can be found by, and the value is the information it hands over when chosen., residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document., self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to., activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer., ALiBiALiBi, abbr.: A simple way to give a transformer a sense of word order: attention between two words is penalized more the farther apart they are, which helps it handle texts longer than those it trained on., attention patternattention pattern, n.: A grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map., attention-based explanationattention-based explanation, n.: Explaining an AI model's output by showing which words or image patches its attention mechanism weighted most — an intuitive but disputed guide to what actually drove the output., causal mediation analysiscausal mediation analysis, n.: A statistical method for splitting a cause's effect into the part that flows through some go-between and the part that does not; in AI it is used to find which internal parts carry a behavior., function vectorfunction vector, n.: A compact internal signal, found inside a language model, that stands for a whole task — such as 'translate into French' or 'give the opposite word' — and can trigger that task when added to the model., grouped-query attentiongrouped-query attention, n.: A middle ground between full multi-head attention and its cheapest variant: attention heads are split into groups, and each group shares one store of information about earlier words, saving memory with little loss in quality., indirect object identificationindirect object identification, n.: A fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model., inference-time interventioninference-time intervention, n.: A technique that makes a language model answer more truthfully by slightly shifting the outputs of a few internal parts that seem to track truth, each time the model produces a word., QK and OV circuitsQK and OV circuits, n.: A way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees.