QK and OV circuits
In plain wordsA way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees.
A decomposition of an attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns. into two independent low-rank bilinear maps. The QK circuit, formed from the query and key weights, determines the attention patternattention pattern, n.: A grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map.: it scores how strongly a token at the destination position attends to each source token. The OV circuit, formed from the value and output weights, determines what the head writes into the residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document. given that it attends to a source token — for example, copying that token's identity toward the output logitslogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity.. Multiplying in the embedding and unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores. gives end-to-end versions that can be read token by token in small models. Because the two maps act separately, a head can be analyzed by asking independently where it attends and what it moves. Composition between heads in different layers is classified by which input of the later head reads the earlier head's output — Q-, K- or V-composition; induction headsinduction head, n.: A pattern-completing mechanism inside language models: having seen 'Harry Potter' once, it spots the next 'Harry' and predicts 'Potter' — a simple trick behind much learning from examples in a prompt. rely on K-composition. Positive eigenvalues of an OV circuit signal copying.
The OV circuit of this head is almost the identity on names, so whatever it attends to gets copied.
Origin
Introduced by Nelson Elhage, Neel Nanda, Catherine Olsson, Tom Henighan, Chris Olah and colleagues in 'A Mathematical Framework for Transformer Circuits' (Anthropic, December 2021), which analyzed attention-only transformers with zero, one and two layers.
Related
- attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.
- attention patternattention pattern, n.: A grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map.
- induction headinduction head, n.: A pattern-completing mechanism inside language models: having seen 'Harry Potter' once, it spots the next 'Harry' and predicts 'Potter' — a simple trick behind much learning from examples in a prompt.
- query, key and valuequery, key and value, n.: The three roles each word plays inside attention: the query is what it is looking for, the key is what it can be found by, and the value is the information it hands over when chosen.
- residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.
- circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
Sources
- Elhage, N., Nanda, N., Olsson, C., Henighan, T. et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread.
Mentioned in
attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns., induction headinduction head, n.: A pattern-completing mechanism inside language models: having seen 'Harry Potter' once, it spots the next 'Harry' and predicts 'Potter' — a simple trick behind much learning from examples in a prompt., query, key and valuequery, key and value, n.: The three roles each word plays inside attention: the query is what it is looking for, the key is what it can be found by, and the value is the information it hands over when chosen., attention patternattention pattern, n.: A grid showing, for each word a language model processes, how much it is looking back at each earlier word at that moment — often drawn as a heat map., Transformer Circuits ThreadTransformer Circuits Thread, prop. n.: Anthropic's online publication for research on how language models work inside, presented as long interactive web articles rather than conventional papers., TransformerLensTransformerLens, prop. n.: A free software library that lets researchers open up language models and record, inspect or alter what happens inside them at every step while the model runs.