direct logit attribution
In plain wordsA quick way to measure how much each internal part of a language model directly pushed it toward a particular next word, by splitting the final score into separate contributions.
A method that attributes a model's output logitlogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity. for a given tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. to individual components by exploiting the additive residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.. Since the final residual state is the sum of the embedding and the outputs of every attention head and MLP layer, projecting each component's output onto the unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores. direction of the target token — after applying the final layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots. scale, frozen at its actual value — gives that component's direct contribution. Using the difference between two tokens' directions (a logit difference, such as 'Mary' minus 'John') isolates the components that decide between them. DLA is cheap, needing a single forward pass, and is a standard first step in circuit analysis. It captures only direct effects: a component that matters by influencing later layers, or whose output is cancelled downstream, is misjudged, so findings are usually confirmed with activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer..
Direct logit attribution pins most of the 'Paris' logit on two heads in the last three layers.
Origin
Follows from the residual-stream decomposition in 'A Mathematical Framework for Transformer Circuits' (Elhage et al., December 2021). The name's first user is not established; it became standard in 2022–2023, spreading largely through Neel Nanda's TransformerLens tutorials and glossary and the circuit studies that built on them.
Related
- residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.
- logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time.
- activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
- unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores.
- indirect object identificationindirect object identification, n.: A fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model.
- attribution patchingattribution patching, n.: A fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part.
Sources
- Elhage, N. et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread.
Mentioned in
residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document., logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time., TransformerLensTransformerLens, prop. n.: A free software library that lets researchers open up language models and record, inspect or alter what happens inside them at every step while the model runs.