distributed alignment search
In plain wordsA method that searches, by training, for a hidden direction inside a neural network that plays the role of one variable in a human-readable explanation of the network's behavior.
A method for finding where a high-level causal variable is represented in a neural network when it is not localized in individual neurons. Instead of patching neurons, DAS learns a rotation of a layer's activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions. and looks for a low-dimensional subspace such that interchange interventions on it — replacing its contents with those computed on another input — change the network's output exactly as intervening on the corresponding variable changes the output of a hypothesized algorithm. The rotation is trained by gradient descent to maximize this agreement (interchange intervention accuracy), combining the rigor of causal abstractioncausal abstraction, n.: A mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped. with the flexibility of learned directions. Boundless DAS scaled the method to models with billions of parameters. Because a learned subspace can produce effects through pathways the model does not normally use, DAS results are vulnerable to interpretability illusionsinterpretability illusion, n.: A convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow. and need careful controls.
DAS found a one-dimensional subspace at layer 12 that behaves exactly like the 'is the price in range' variable.
Origin
Introduced by Atticus Geiger, Zhengxuan Wu, Christopher Potts, Thomas Icard and Noah Goodman in 'Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations' (arXiv, March 2023; CLeaR 2024). Zhengxuan Wu and colleagues' Boundless DAS (NeurIPS 2023) applied it to the Alpaca model.
Usage note
'Alignment' here means matching neural representations to variables of an interpretable model, not AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this..
Related
- causal abstractioncausal abstraction, n.: A mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped.
- activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
- interpretability illusioninterpretability illusion, n.: A convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow.
- linear representation hypothesislinear representation hypothesis, n.: The idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction.
- activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions.
Sources
- Geiger, A., Wu, Z., Potts, C., Icard, T. & Goodman, N. (2024). Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations. CLeaR 2024.
- Makelov, A., Lange, G. & Nanda, N. (2024). Is This the Subspace You Are Looking For? An Interpretability Illusion for Subspace Activation Patching. ICLR 2024.
Mentioned in
causal abstractioncausal abstraction, n.: A mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped., interpretability illusioninterpretability illusion, n.: A convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow.