mechanistic interpretability
In plain wordsA branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
The subfield of interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box. that tries to identify the algorithm a trained neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain. actually implements, described in terms of its internal parts; Chris Olah compares the project to recovering readable source code from a compiled binary. It looks for human-understandable featuresfeature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity. — directions in activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions. that represent concepts — and circuitscircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name., the weighted connections through which features compute one another, and it tests hypotheses causally by intervening on activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. with ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network. and activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.. Landmark results include induction headsinduction head, n.: A pattern-completing mechanism inside language models: having seen 'Harry Potter' once, it spots the next 'Harry' and predicts 'Potter' — a simple trick behind much learning from examples in a prompt., the indirect object identificationindirect object identification, n.: A fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model. circuit in GPT-2GPT-2, prop. n.: A 2019 OpenAI language model that wrote surprisingly fluent paragraphs. OpenAI at first withheld the full version, saying it could be misused for fake news, which made it famous., the analysis of superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read., the use of sparse autoencoderssparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. to extract millions of largely monosemanticmonosemanticity, n.: The property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several. features from production models, and attribution graphsattribution graph, n.: A diagram showing which internal concepts led an AI model to a particular answer, with arrows showing how strongly each concept pushed the next. that trace multi-step reasoning in ClaudeClaude, prop. n.: A family of AI assistants and language models made by the company Anthropic since 2023, widely used for writing, analysis and computer programming.. It differs from input-attribution methods in seeking mechanisms rather than correlations, and is pursued both as basic science (model biologymodel biology, n.: The study of what AI models actually do inside — the strategies and hidden steps they use — approached the way biologists study living things, by observation and experiment rather than from a blueprint.) and as a possible tool for auditing alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., for example by detecting deception or hidden goals.
Our mech interp team traced the refusal back to a handful of features in the middle layers.
Origin
Credited to Chris Olah, who has said he chose it to set the reverse-engineering of circuits apart from saliency-map methods; Naomi Saphra and Sarah Wiegreffe (2024) trace its first public uses to the Distill Circuits thread written by Olah's team at OpenAI (March 2020–April 2021). It spread through Anthropic's Transformer Circuits Thread (from December 2021) and Neel Nanda's tutorials, open-problem lists and TransformerLens library (2022).
Usage note
The boundaries are disputed. Naomi Saphra and Sarah Wiegreffe (2024) distinguish narrow technical use (causal mechanisms linking inputs to outputs), broad technical use (any study of activations or weights), and cultural uses naming a particular research community or, loosely, any interpretability work on language models.
Related
- interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box.
- circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
- feature (interpretability)feature (interpretability), n.: A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity.
- superpositionsuperposition, n.: The way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read.
- sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
- activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
- model biologymodel biology, n.: The study of what AI models actually do inside — the strategies and hidden steps they use — approached the way biologists study living things, by observation and experiment rather than from a blueprint.
- Chris OlahChris Olah, prop. n.: A Canadian researcher who pioneered ways of looking inside neural networks to see what they have learned, and a co-founder of the AI company Anthropic.
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
- explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application.
Sources
- Olah, C., Cammarata, N., Schubert, L., Goh, G., Petrov, M. & Carter, S. (2020). Zoom In: An Introduction to Circuits. Distill. distill.pub/2020/circuits/zoom-in
- Elhage, N. et al. (2021). A Mathematical Framework for Transformer Circuits. Transformer Circuits Thread.
- Saphra, N. & Wiegreffe, S. (2024). Mechanistic? Proceedings of BlackboxNLP 2024. arxiv.org/html/2410.09087v1
Mentioned in
AnthropicAnthropic, prop. n.: An American AI company that makes the Claude chatbot and models. It was founded in 2021 by former OpenAI staff and puts particular emphasis on AI safety research., explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application., interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., black boxblack box, n.: A system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer., causal inferencecausal inference, n.: Working out whether one thing actually causes another, rather than merely occurring alongside it — for example, whether a new feature made users stay longer or just happened at the same time., Chris OlahChris Olah, prop. n.: A Canadian researcher who pioneered ways of looking inside neural networks to see what they have learned, and a co-founder of the AI company Anthropic., circuitcircuit, n.: A small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name., feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down., global explanationglobal explanation, n.: An explanation of how an AI model behaves overall — which factors matter most across all its decisions, and in what way — rather than why it made one particular decision., GPT-2GPT-2, prop. n.: A 2019 OpenAI language model that wrote surprisingly fluent paragraphs. OpenAI at first withheld the full version, saying it could be misused for fake news, which made it famous., induction headinduction head, n.: A pattern-completing mechanism inside language models: having seen 'Harry Potter' once, it spots the next 'Harry' and predicts 'Potter' — a simple trick behind much learning from examples in a prompt., machine understandingmachine understanding, n.: Whether, and in what sense, an AI system grasps what words, situations or problems mean — rather than just producing answers that look as though it does., Max TegmarkMax Tegmark, prop. n.: A Swedish-American physicist at MIT who co-founded the Future of Life Institute, which organized the 2023 open letter calling for a pause in training the most powerful AI systems., treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it., developmental interpretabilitydevelopmental interpretability, n.: The study of how a neural network's internal structure forms during training — when and how new abilities and mechanisms appear — rather than only examining the finished model., DistillDistill, prop. n.: An online scientific journal (2016–2021) that published clear, interactive explanations of machine learning research, including founding work on understanding what happens inside neural networks., feature visualizationfeature visualization, n.: Making pictures that show what a part of an image-recognition network is looking for, by repeatedly adjusting an image until that part responds as strongly as possible., GoodfireGoodfire, prop. n.: A San Francisco company, founded in 2024, that builds tools to look inside AI models, understand what they have learned and adjust how they behave., grokkinggrokking, n.: When a model that has merely memorized its training examples suddenly, long afterward, figures out the underlying rule and starts getting new examples right., indirect object identificationindirect object identification, n.: A fill-in-the-name task — 'When Mary and John went to the store, John gave a drink to …' — whose solution inside GPT-2 became the best-known fully mapped circuit in a language model., machine psychologymachine psychology, n.: Studying AI systems the way psychologists study people — with carefully designed experiments on their behavior — to discover what they can do, how they reason and where they go wrong., Marr's levels of analysisMarr's levels of analysis, n.: A way of explaining any information-processing system on three levels: what problem it solves and why, what step-by-step method it uses, and how that method is physically built., mechanistic anomaly detectionmechanistic anomaly detection, n.: Flagging cases where an AI reaches its output by an unusual internal route — for instance because it is being tricked, or is tricking us — even when the output itself looks normal., model biologymodel biology, n.: The study of what AI models actually do inside — the strategies and hidden steps they use — approached the way biologists study living things, by observation and experiment rather than from a blueprint.