replacement model
In plain wordsA more understandable copy of an AI model, built from simple labeled parts, that behaves almost like the original, so researchers can study the copy's workings instead.
An interpretable surrogate in which some of a model's components are swapped for sparse dictionary models — typically cross-layer transcoderstranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes. standing in for every MLP layer — so that computation flows through human-interpretable featuresfeature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity.. Its fidelity is measured by how often it reproduces the original model's next-token predictions. A local replacement model, built for one prompt, additionally freezes the original's attention patterns and normalization denominators and adds error nodes holding the difference between original and replacement outputs at each layer and position, so its output matches the original's exactly on that prompt; attribution graphsattribution graph, n.: A diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next. are computed from it. The approach assumes that the replacement's mechanisms resemble the original's, an assumption checked by intervention experiments on the original; where error nodes carry much of the effect, the explanation is incomplete.
On this prompt the error nodes carry a third of the effect, so the replacement model's story is only partial.
Origin
Introduced in Anthropic's 'Circuit Tracing: Revealing Computational Graphs in Language Models' (Emmanuel Ameisen, Jack Lindsey and colleagues, March 2025). Studying a model through an interpretable stand-in echoes the surrogate modelssurrogate model, n.: A simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply. of explainable AI, though here the stand-in is trained to match internal computations, not just outputs.
Related
- circuit tracingcircuit tracing, n.: Methods that map, step by step, which internal concepts an AI model used to produce a particular answer and how they fed into one another, like tracing wires on a circuit board.
- attribution graphattribution graph, n.: A diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next.
- transcodertranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes.
- interpretability illusioninterpretability illusion, n.: A convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow.
- dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts.
- surrogate modelsurrogate model, n.: A simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply.
Sources
- Ameisen, E., Lindsey, J., Pearce, A. et al. (2025). Circuit Tracing: Revealing Computational Graphs in Language Models. Transformer Circuits Thread.
Mentioned in
attribution graphattribution graph, n.: A diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next., circuit tracingcircuit tracing, n.: Methods that map, step by step, which internal concepts an AI model used to produce a particular answer and how they fed into one another, like tracing wires on a circuit board., transcodertranscoder, n.: A stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes.