Patchscopes
In plain wordsA family of tricks for getting a language model to describe its own hidden states in words, by transplanting an internal state into a new prompt designed to make the model talk about it.
A framework for decoding the information in a language model's hidden representations by patching a representation from one forward pass (the source) into a different forward pass (the target) whose prompt is designed to make the model put it into words. For example, the hidden state of a subject token can be inserted into a few-shot identity prompt such as 'cat → cat; 1135 → 1135; hello → hello; ?' at the position of the question mark, so that the model generates a description of the entity the state encodes. The choice of source and target layers, positions, prompt and even model defines a particular Patchscope; the logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time., tuned lenstuned lens, n.: An improved version of the logit lens that first learns a small translator for each layer, so a language model's halfway-through states can be read as predictions more accurately. and several probing methods are special cases. Patchscopes have been used to follow how entities are resolved across layers, to diagnose multi-hop reasoning failures and, by patching into a larger model, to have one model explain another's representations. Outputs can reflect the target prompt as well as the source.
The Patchscope decodes the layer-3 state for 'Diana' as 'Princess of Wales' by layer 6.
Origin
Introduced by Asma Ghandeharioun, Avi Caciularu, Adam Pearce, Lucas Dixon and Mor Geva of Google in 'Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models' (January 2024; ICML 2024). Related self-interpretation methods include Future Lens (2023) and SelfIE (2024).
Related
- logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time.
- tuned lenstuned lens, n.: An improved version of the logit lens that first learns a small translator for each layer, so a language model's halfway-through states can be read as predictions more accurately.
- activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
- probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere.
- automated interpretabilityautomated interpretability, n.: Using AI models to help explain other AI models — for instance, having a language model read examples that excite an internal unit and write a description of what it detects.
Sources
- Ghandeharioun, A., Caciularu, A., Pearce, A., Dixon, L. & Geva, M. (2024). Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models. ICML 2024. arXiv:2401.06102. arxiv.org/abs/2401.06102
Mentioned in
logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time., tuned lenstuned lens, n.: An improved version of the logit lens that first learns a small translator for each layer, so a language model's halfway-through states can be read as predictions more accurately.