activation patching
In plain wordsAn experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
A causal intervention in which the activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. of a chosen component — a layer, attention headattention head, n.: One of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns., token position or feature — recorded on one input are substituted into the model's forward pass on a different input, and the change in output is measured. Typically a clean prompt and a corrupted prompt differing in one key detail are used. Patching clean activations into the corrupted run (denoising) tests which components suffice to restore the behavior; patching corrupted activations into the clean run (noising) tests which are necessary. Effects are measured by metrics such as logit difference or the probability of the correct answer, and sweeping over layers and positions localizes where information is processed. Results depend on the choice of corruption (substituted prompts versus added noise) and of metric, and patching learned subspaces rather than whole components can produce illusoryinterpretability illusion, n.: A convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow. effects. attribution patchingattribution patching, n.: A fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part. approximates it cheaply with gradients.
Patching the subject position at layers 10 to 15 restores most of the correct answer's probability.
Origin
Grew out of causal mediation analysis of language models by Jesse Vig and colleagues (2020), interchange interventions in Atticus Geiger and colleagues' causal-abstraction work (2021) and the causal tracing of Kevin Meng and colleagues (2022). The name, whose first user is not established, was popularized from 2022 by Neel Nanda and others in mechanistic interpretability; Fred Zhang and Nanda (ICLR 2024) and Stefan Heimersheim and Nanda (2024) codified best practices.
Related
- causal tracingcausal tracing, n.: A test that blurs the words of a factual question with noise and then restores one internal part of the model at a time, to find where in the model the fact gets recalled.
- path patchingpath patching, n.: A refined patching experiment that changes only one route of information inside a model — from one part to another — to test exactly which connections carry a behavior.
- attribution patchingattribution patching, n.: A fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part.
- ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network.
- causal mediation analysiscausal mediation analysis, n.: A statistical method for splitting a cause's effect into the part that flows through some go-between and the part that does not; in AI it is used to find which internal parts carry a behavior.
- causal abstractioncausal abstraction, n.: A mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped.
- circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
Sources
- Zhang, F. & Nanda, N. (2024). Towards Best Practices of Activation Patching in Language Models: Metrics and Methods. ICLR 2024.
- Heimersheim, S. & Nanda, N. (2024). How to use and interpret activation patching. arXiv preprint.
Mentioned in
interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior., ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network., activationactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., activation vectoractivation vector, n.: The list of numbers a neural network produces at one layer for one input — a snapshot of its internal state. In steering, also a direction added to that state to shift behavior., causal inferencecausal inference, n.: Working out whether one thing actually causes another, rather than merely occurring alongside it — for example, whether a new feature made users stay longer or just happened at the same time., circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name., counterfactualcounterfactual, n.: A statement about what would have happened if something had been different — 'had I left earlier, I would have caught the train'. AI uses counterfactuals to explain decisions and to measure causes., probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere., attention-based explanationattention-based explanation, n.: Explaining an AI model's output by showing which words or image patches its attention mechanism weighted most — an intuitive but disputed guide to what actually drove the output., attribution patchingattribution patching, n.: A fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part., automated circuit discoveryautomated circuit discovery, n.: Software methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand., causal abstractioncausal abstraction, n.: A mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped., causal mediation analysiscausal mediation analysis, n.: A statistical method for splitting a cause's effect into the part that flows through some go-between and the part that does not; in AI it is used to find which internal parts carry a behavior., causal scrubbingcausal scrubbing, n.: A strict test of an explanation of how a model works: scramble every internal signal the explanation says should not matter, and check that the model still performs as before., causal tracingcausal tracing, n.: A test that blurs the words of a factual question with noise and then restores one internal part of the model at a time, to find where in the model the fact gets recalled., direct logit attributiondirect logit attribution, n.: A quick way to measure how much each internal part of a language model directly pushed it toward a particular next word, by splitting the final score into separate contributions., distributed alignment searchdistributed alignment search, n.: A method that searches, by training, for a hidden direction inside a neural network that plays the role of one variable in a human-readable explanation of the network's behavior., function vectorfunction vector, n.: A compact internal signal, found inside a language model, that stands for a whole task — such as 'translate into French' or 'give the opposite word' — and can trigger that task when added to the model., interpretability illusioninterpretability illusion, n.: A convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow., model-agnosticmodel-agnostic, adj.: Describes a method that works with any kind of AI model, because it only needs to feed the model inputs and observe its outputs, never to look inside., Neel NandaNeel Nanda, prop. n.: A British AI researcher who leads mechanistic interpretability at Google DeepMind and built TransformerLens, a widely used tool for looking inside language models., PatchscopesPatchscopes, prop. n.: A family of tricks for getting a language model to describe its own hidden states in words, by transplanting an internal state into a new prompt designed to make the model talk about it., path patchingpath patching, n.: A refined patching experiment that changes only one route of information inside a model — from one part to another — to test exactly which connections carry a behavior.