logit lens
In plain wordsA quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time.
A technique that decodes intermediate residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document. states of a transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. language model directly into vocabulary space by applying the model's own final layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots. and unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores. to them, as if each layer were the last. Plotting the resulting next-token distributions layer by layer shows how a prediction forms: in GPT-2, plausible guesses often appear in the middle layers and are refined toward the final answer, suggesting that the model works in an output-like space throughout. The method needs no training and is a standard first look at a new model. It is unreliable in early layers and on some model families, whose intermediate states are not in the basis the unembedding expects; the tuned lenstuned lens, n.: An improved version of the logit lens that first learns a small translator for each layer, so a language model's halfway-through states can be read as predictions more accurately. corrects this with a learned translator for each layer. Chris Wendler and colleagues (2024) used logit-lens readings to argue that Llama-2 models process non-English prompts through an English-leaning intermediate representation.
Under the logit lens, 'Paris' becomes the top token by layer 18 and stays there.
Origin
Introduced by the pseudonymous blogger nostalgebraist in the LessWrong post 'interpreting GPT: the logit lens' (August 2020), which applied it to GPT-2.
Related
- tuned lenstuned lens, n.: An improved version of the logit lens that first learns a small translator for each layer, so a language model's halfway-through states can be read as predictions more accurately.
- residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.
- unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores.
- direct logit attributiondirect logit attribution, n.: A quick way to measure how much each internal part of a language model directly pushed it toward a particular next word, by splitting the final score into separate contributions.
- PatchscopesPatchscopes, prop. n.: A family of tricks for getting a language model to describe its own hidden states in words, by transplanting an internal state into a new prompt designed to make the model talk about it.
- probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere.
Sources
- nostalgebraist (2020). interpreting GPT: the logit lens. LessWrong.
- Wendler, C., Veselovsky, V., Monea, G. & West, R. (2024). Do Llamas Work in English? On the Latent Language of Multilingual Transformers. ACL 2024.
Mentioned in
logitlogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity., residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document., direct logit attributiondirect logit attribution, n.: A quick way to measure how much each internal part of a language model directly pushed it toward a particular next word, by splitting the final score into separate contributions., PatchscopesPatchscopes, prop. n.: A family of tricks for getting a language model to describe its own hidden states in words, by transplanting an internal state into a new prompt designed to make the model talk about it., seahorse emojiseahorse emoji, n.: A viral meme in which chatbots, asked whether a seahorse emoji exists, insisted it did and then spiraled through wrong emojis trying to produce one — though there was no such emoji., tuned lenstuned lens, n.: An improved version of the logit lens that first learns a small translator for each layer, so a language model's halfway-through states can be read as predictions more accurately., unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores.