attention pattern
In plain wordsA grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map.
The matrix of attention weights produced by one attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns. on one input: entry (i, j) is the softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.-normalized score with which the query at position i attends to the key at position j, so each row sums to one and, in models with a causal maskcausal mask, n.: A rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat., entries above the diagonal are zero. Attention patterns show where each head gathers information from and are drawn as heat maps or arc diagrams. Recognizable patterns identify head types: previous-token heads (a stripe just below the diagonal), induction headsinduction head, n.: A pattern-completing mechanism inside language models: having seen 'Harry Potter' once, it spots the next 'Harry' and predicts 'Potter' — a simple trick behind much learning from examples in a prompt. (an offset stripe on repeated text), heads that park attention on the first token as a default attention sinkattention sink, n.: A quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need., and heads tracking syntactic relations. In circuit analysis the pattern, governed by the QK circuitQK and OV circuits, n.: A way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees., is studied separately from what the head writes, governed by the OV circuit. A pattern shows where a head looks, not what it moves or how much that matters to the output.
Layer 5 head 1 has a clean induction stripe in its attention pattern on the repeated half.
Origin
Plots of attention weights date to neural machine translation, where Dzmitry Bahdanau, Kyunghyun Cho and Yoshua Bengio (2014) displayed them as soft alignments. Kevin Clark and colleagues' 'What Does BERT Look At?' and Jesse Vig's BertViz tool (both 2019) made head-by-head analysis routine; 'attention pattern' is the standard mechanistic-interpretability term since Elhage et al. (2021).
Usage note
Whether attention weights explain predictions was debated in 'Attention is not Explanation' (Sarthak Jain and Byron Wallace, 2019) and the reply 'Attention is not not Explanation' (Sarah Wiegreffe and Yuval Pinter, 2019); see attention-based explanationattention-based explanation, n.: Explaining an AI model's output by showing which words or image patches its attention mechanism weighted most — an intuitive but disputed guide to what actually drove the output..
Related
- attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones.
- attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.
- QK and OV circuitsQK and OV circuits, n.: A way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees.
- induction headinduction head, n.: A pattern-completing mechanism inside language models: having seen 'Harry Potter' once, it spots the next 'Harry' and predicts 'Potter' — a simple trick behind much learning from examples in a prompt.
- attention sinkattention sink, n.: A quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need.
- attention-based explanationattention-based explanation, n.: Explaining an AI model's output by showing which words or image patches its attention mechanism weighted most — an intuitive but disputed guide to what actually drove the output.
Sources
- Clark, K., Khandelwal, U., Levy, O. & Manning, C. D. (2019). What Does BERT Look At? An Analysis of BERT's Attention. BlackboxNLP 2019.
- Vig, J. (2019). A Multiscale Visualization of Attention in the Transformer Model. ACL 2019 System Demonstrations.
Mentioned in
attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones., attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns., induction headinduction head, n.: A pattern-completing mechanism inside language models: having seen 'Harry Potter' once, it spots the next 'Harry' and predicts 'Potter' — a simple trick behind much learning from examples in a prompt., attention-based explanationattention-based explanation, n.: Explaining an AI model's output by showing which words or image patches its attention mechanism weighted most — an intuitive but disputed guide to what actually drove the output., QK and OV circuitsQK and OV circuits, n.: A way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees.