causal scrubbing
In plain wordsA strict test of an explanation of how a model works: scramble every internal signal the explanation says should not matter, and check that the model still performs as before.
A method for evaluating a mechanistic hypothesis by systematic resampling ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network.. The hypothesis is stated as a correspondence between the model's computational graph and an interpretable computation, which implies that certain activations should be interchangeable: inputs that the hypothesis treats as equivalent at a given node should be swappable there without changing the output. Causal scrubbing performs all such swaps at once — replacing activations with ones computed on other inputs that agree on everything the hypothesis says matters — and measures how much of the model's performance, such as its loss on a task, is preserved. A correct and complete hypothesis should preserve nearly all of it; the fraction lost quantifies what the explanation misses. Its strictness exposed gaps in earlier circuit explanations, but it can be fooled by errors that cancel and becomes expensive for detailed hypotheses.
Our induction hypothesis recovers only 72% of the loss under causal scrubbing, so something is missing.
Origin
Introduced by Lawrence Chan, Adrià Garriga-Alonso, Nicholas Goldowsky-Dill, Ryan Greenblatt, Jenny Nitishinskaya, Ansh Radhakrishnan, Buck Shlegeris and Nate Thomas of Redwood Research in a series of AI Alignment Forum posts (December 2022).
Related
- activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
- ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network.
- causal abstractioncausal abstraction, n.: A mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped.
- circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
- path patchingpath patching, n.: A refined patching experiment that changes only one route of information inside a model — from one part to another — to test exactly which connections carry a behavior.
- interpretability illusioninterpretability illusion, n.: A convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow.
Sources
- Chan, L., Garriga-Alonso, A., Goldowsky-Dill, N., Greenblatt, R., Nitishinskaya, J., Radhakrishnan, A., Shlegeris, B. & Thomas, N. (2022). Causal Scrubbing: a method for rigorously testing interpretability hypotheses. AI Alignment Forum.
Mentioned in
ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network., circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name., Buck ShlegerisBuck Shlegeris, prop. n.: An AI-safety researcher who runs Redwood Research and helped found 'AI control', the study of keeping powerful AI safe even if it is secretly trying to cause harm., causal abstractioncausal abstraction, n.: A mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped., interpretability illusioninterpretability illusion, n.: A convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow., path patchingpath patching, n.: A refined patching experiment that changes only one route of information inside a model — from one part to another — to test exactly which connections carry a behavior.