path patching
In plain wordsA refined patching experiment that changes only one route of information inside a model — from one part to another — to test exactly which connections carry a behavior.
A generalization of activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer. that intervenes on paths — specific routes by which one component affects another — rather than on whole components. To test whether head A influences the output through head B, the corrupted activation of A is propagated only along the edge into B, with all other paths kept at their clean values, and the effect on the output is measured. Path patching separates direct effects from effects mediated by particular downstream components, letting researchers assemble a circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name. edge by edge; it was the main tool behind the indirect object identificationindirect object identification, n.: A fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model. circuit. Nicholas Goldowsky-Dill and colleagues formalized it as a way to test hypotheses expressed as sets of paths through the computational graph. Its cost grows with the number of paths examined, motivating automated circuit discoveryautomated circuit discovery, n.: Software methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand. and gradient-based approximations.
Path patching shows the S-inhibition heads matter only through the queries of the name-mover heads.
Origin
Introduced by Kevin Wang and colleagues in the indirect-object-identification study (November 2022) and formalized by Nicholas Goldowsky-Dill, Chris MacLeod, Lucas Sato and Aryaman Arora in 'Localizing Model Behavior with Path Patching' (2023), both from work at Redwood Research.
Related
- activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
- indirect object identificationindirect object identification, n.: A fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model.
- circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
- causal scrubbingcausal scrubbing, n.: A strict test of an explanation of how a model works: scramble every internal signal the explanation says should not matter, and check that the model still performs as before.
- automated circuit discoveryautomated circuit discovery, n.: Software methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand.
- attribution patchingattribution patching, n.: A fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part.
Sources
- Goldowsky-Dill, N., MacLeod, C., Sato, L. & Arora, A. (2023). Localizing Model Behavior with Path Patching. arXiv preprint.
- Wang, K., Variengien, A., Conmy, A., Shlegeris, B. & Steinhardt, J. (2023). Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small. ICLR 2023.
Mentioned in
circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name., activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer., attribution patchingattribution patching, n.: A fast shortcut that estimates the results of thousands of patching experiments at once, using the slope of the model's output instead of rerunning the model for every part., automated circuit discoveryautomated circuit discovery, n.: Software methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand., causal abstractioncausal abstraction, n.: A mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped., causal scrubbingcausal scrubbing, n.: A strict test of an explanation of how a model works: scramble every internal signal the explanation says should not matter, and check that the model still performs as before., indirect object identificationindirect object identification, n.: A fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model.