probing
In plain wordsTesting what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere.
A family of methods that test whether information is encoded in a network's activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. by training a simple supervised model, the probe, to predict a property of interest — part of speech, syntactic depth, truth value, a board position, the presence of a concept — from those activations, usually with the network frozen. High probe accuracy shows that the property is decodable from the representation; comparing layers shows where it emerges. Probing was a central method in the analysis of BERTBERT, prop. n.: An influential 2018 Google language model that learned to understand text by filling in hidden words. It made search engines and many language tools much better at grasping meaning in context.-era models and remains widely used, including for monitoring models for deception or harmful intent. Its limits are interpretive. A powerful probe can learn the task itself, so researchers prefer simple probes and use control tasks and selectivity measures; and decodability does not show that the model uses the information, which requires causal tests such as activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer., ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network. or concept erasureconcept erasure, n.: Removing a specific idea — such as a person's gender, or an artist's style — from what an AI model can represent or produce, so the model can no longer use or show it..
Probing showed that syntactic depth is most decodable around the middle layers.
Origin
Guillaume Alain and Yoshua Bengio introduced 'linear classifier probes' (arXiv, October 2016); in the same period Yossi Adi and colleagues used 'auxiliary prediction tasks' (2016) and Dieuwke Hupkes and colleagues 'diagnostic classifiers' (2017–2018). Alexis Conneau and colleagues' 'probing tasks' (2018) and John Hewitt's structural probes and control tasks (2019) consolidated the method.
Usage note
Probing is correlational: it shows what can be read out, not what the model reads. Yonatan Belinkov's survey 'Probing Classifiers: Promises, Shortcomings, and Advances' (2022) reviews the debates.
Related
- linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly.
- activation patchingactivation patching, n.: An experiment that copies part of a model's internal state from its run on one input into its run on another, to see which internal parts actually cause a change in the answer.
- concept erasureconcept erasure, n.: Removing a specific idea — such as a person's gender, or an artist's style — from what an AI model can represent or produce, so the model can no longer use or show it.
- testing with concept activation vectorstesting with concept activation vectors, phr.: A method for asking whether a human idea, such as 'wheels', matters to an image model's decision, such as 'car', by finding the direction for that idea inside the model and measuring its effect.
- interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box.
- Othello-GPTOthello-GPT, prop. n.: A small language model trained only on lists of moves from the board game Othello, which researchers found had built an internal picture of the board — evidence that such models can form world models.
Sources
- Alain, G. & Bengio, Y. (2016). Understanding intermediate layers using linear classifier probes. arXiv:1610.01644. arxiv.org/abs/1610.01644
- Hewitt, J. & Liang, P. (2019). Designing and Interpreting Probes with Control Tasks. EMNLP 2019.
- Belinkov, Y. (2022). Probing Classifiers: Promises, Shortcomings, and Advances. Computational Linguistics 48(1).
Mentioned in
interpretabilityinterpretability, n.: The effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box., activationactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., activation vectoractivation vector, n.: The list of numbers a neural network produces at one layer for one input — a snapshot of its internal state. In steering, also a direction added to that state to shift behavior., linear probelinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly., machine understandingmachine understanding, n.: Whether, and in what sense, an AI system grasps what words, situations or problems mean — rather than just producing answers that look as though it does., post hoc explanationpost hoc explanation, n.: An explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable., steeringsteering, n.: Guiding what an AI model does — its topic, tone or behavior — while it runs, by changing its instructions, its internal signals or how its next word is chosen, rather than by retraining it., alignment auditalignment audit, n.: A systematic investigation of an AI model to find out whether it has hidden goals or bad tendencies — such as deceiving its users — that ordinary testing might miss., contrast-consistent searchcontrast-consistent search, n.: A method for finding where a language model internally tracks whether statements are true, using only the logical rule that a statement and its opposite cannot both be true., directional ablationdirectional ablation, n.: Erasing one specific direction from an AI model's internal signals while it runs, so whatever that direction encodes — such as the tendency to refuse — can no longer influence what the model does., eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe., interpretability illusioninterpretability illusion, n.: A convincing-looking explanation of what is going on inside an AI model that turns out to be wrong or misleading, often because the evidence examined was too narrow., iterative nullspace projectioniterative nullspace projection, n.: An early method for removing a trait such as gender from a model's internal representations by repeatedly training a simple detector for it and then flattening away whatever the detector relied on., logit lenslogit lens, n.: A quick trick for peeking at what a language model is 'thinking' partway through: read its half-finished internal state as if it were the final answer, one layer at a time., model-agnosticmodel-agnostic, adj.: Describes a method that works with any kind of AI model, because it only needs to feed the model inputs and observe its outputs, never to look inside., Othello-GPTOthello-GPT, prop. n.: A small language model trained only on lists of moves from the board game Othello, which researchers found had built an internal picture of the board — evidence that such models can form world models., PatchscopesPatchscopes, prop. n.: A family of tricks for getting a language model to describe its own hidden states in words, by transplanting an internal state into a new prompt designed to make the model talk about it., tuned lenstuned lens, n.: An improved version of the logit lens that first learns a small translator for each layer, so a language model's halfway-through states can be read as predictions more accurately., uncertainty quantificationuncertainty quantification, n.: Measuring how unsure an AI system's predictions are, and why, so that people know when to trust an answer and when to double-check it.