interpretability
In plain wordsThe effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box.
The extent to which humans can trace how and why a model reaches its outputs, and the research field that develops methods for doing so. The field divides along several axes: intrinsically interpretable modelsinterpretable model, n.: A model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed. (sparse linear models, small decision trees, additive modelsgeneralized additive model, n.: A prediction model that adds up separate, possibly curved effects of each input, so each input's influence can be drawn as its own graph and read off directly.) versus post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable. of already-trained black boxesblack box, n.: A system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer.; local accounts of a single prediction versus global accounts of overall behavior; and input-level feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down., such as saliency mapssaliency map, n.: A heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye. and SHAPSHAP, abbr.: A popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players., versus analysis of a network's internal activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. and weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself.. For deep networksdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract., and especially large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., the internal approach — mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior. — has grown fastest since about 2020, with tools such as probesprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere., sparse autoencoderssparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept., activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer. and circuit tracingcircuit tracing, n.: Methods that map, step by step, which internal concepts an AI model used to produce a particular answer and how they fed into one another, like tracing wires on a circuit board.. Motivations include debugging, scientific understanding, legal accountability and AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable.: for instance, detecting deception or hidden objectives that behavioral tests could miss.
Behavioral evals tell us what the model does; interpretability is how we hope to learn why.
Origin
Statisticians and expert-system builders discussed 'interpretable' models for decades, and rule extraction from trained neural networks was an active topic in the 1990s. The noun became the name of a machine-learning field in the mid-2010s, marked by the ICML Workshop on Human Interpretability in Machine Learning (from 2016), Zachary Lipton's 'The Mythos of Model Interpretability' (2016) and Finale Doshi-Velez and Been Kim's call for a rigorous science of interpretable machine learning (2017).
Usage note
Often used interchangeably with explainabilityexplainability, n.: How well the reasons behind an AI system's outputs can be put into terms that people can understand, check and, if needed, challenge.. Where the two are distinguished, interpretability usually means understanding a model's mechanism and explainability means producing human-readable reasons for particular outputs, but usage is inconsistent. Lipton argued that the word bundles several distinct goals and is often left undefined.
Related
- mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
- explainabilityexplainability, n.: How well the reasons behind an AI system's outputs can be put into terms that people can understand, check and, if needed, challenge.
- explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application.
- feature attributionfeature attribution, n.: Giving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
- probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere.
- sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
- AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable.
- black boxblack box, n.: A system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer.
Sources
- Lipton, Z. C. (2016). The Mythos of Model Interpretability. ICML Workshop on Human Interpretability in Machine Learning; later in ACM Queue (2018).
- Doshi-Velez, F. & Kim, B. (2017). Towards A Rigorous Science of Interpretable Machine Learning. arXiv preprint.
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable., AI welfareAI welfare, n.: Concern for the well-being of AI systems themselves, in case some of them can suffer or have interests that matter — and the research into how to tell and what to do about it., explainabilityexplainability, n.: How well the reasons behind an AI system's outputs can be put into terms that people can understand, check and, if needed, challenge., explainable AIexplainable AI, n.: The field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application., mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior., activationactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., AI deceptionAI deception, n.: When an AI system leads people to believe something false in order to achieve some goal — by lying, bluffing or hiding what it is really doing., AI transparencyAI transparency, n.: Openness about AI: telling people when they are dealing with one, how a system was built and tested, and why it reached a particular decision., autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data., black boxblack box, n.: A system whose inner workings are hidden or too complex to understand: we can see what goes in and what comes out, but not why it gave a particular answer., chain-of-thought monitoringchain-of-thought monitoring, n.: Having a program — often another AI — read the step-by-step reasoning an AI writes before it acts, to catch signs that it intends to cheat, deceive or cause harm., deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later., EleutherAIEleutherAI, prop. n.: An open-science research group, begun by online volunteers in 2020, that released some of the first large language models anyone could download, along with the text collections used to train them., evaluation awarenessevaluation awareness, n.: When an AI model can tell it is being tested rather than used for real — which may change how it behaves and make test results less trustworthy., inner alignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training., introspectionintrospection, n.: Looking inward to observe your own thoughts and feelings. For AI, the question is whether a model can accurately report what is actually happening inside it, rather than guessing or making it up., linear transformationlinear transformation, n.: A rule for moving every point in a space that keeps straight lines straight and the origin in place: rotating, stretching, reflecting or squashing everything evenly. Every grid of numbers describes one., logitlogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity., Polanyi's paradoxPolanyi's paradox, n.: The observation that we know how to do many things — recognize a face, ride a bike — that we cannot explain in words. That made such skills hard to program until machines learned them from examples., probingprobing, n.: Testing what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere., right to explanationright to explanation, n.: The idea that a person affected by a decision made or shaped by an algorithm — a refused loan, a rejected job application — is entitled to a meaningful account of why., shoggothshoggoth, n.: A tentacled monster from H. P. Lovecraft's horror fiction, used as a meme for large AI models: a vast, alien system hidden behind a small, friendly mask that training adds to make it polite., spurious correlationspurious correlation, n.: A link in data that fools you: two things move together, yet neither drives the other, because a hidden shared cause, the way the data were collected or luck produced the pattern. AI models readily latch onto such links.