mechanistic anomaly detection
In plain wordsFlagging cases where an AI reaches its output by an unusual internal route — for instance because it is being tricked, or is tricking us — even when the output itself looks normal.
The task of detecting, from a model's internal computation, inputs on which it produces its output for an unusual reason, compared with a trusted reference set of inputs on which it behaves as intended. If a model normally predicts 'the diamond is safe' because the diamond is there, a detector should flag a case where the same prediction arises because the sensors were tampered with, even without a full understanding of either mechanism. Approaches compare activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., attributions or other internal statistics against the trusted distribution. Intended applications include detecting backdoorsbackdoor attack, n.: Secretly training an AI model to misbehave whenever it sees a hidden signal — a sticker, a word, a date — while it acts perfectly normally the rest of the time., measurement tampering and treacherous behavior that appears only in deployment. It differs from ordinary anomaly detectionanomaly detection, n.: Automatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network., which flags unusual inputs or outputs, by targeting unusual mechanisms.
Mechanistic anomaly detection flagged the episode even though the agent's final report looked routine.
Origin
Proposed by Paul Christiano and the Alignment Research Center in 2022 as an approach to eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe., linked to their work on heuristic explanations. Redwood Research's measurement-tampering benchmarks (Fabien Roger and colleagues, 2023) and later EleutherAI experiments tested practical detectors.
Related
- eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe.
- anomaly detectionanomaly detection, n.: Automatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network.
- backdoor attackbackdoor attack, n.: Secretly training an AI model to misbehave whenever it sees a hidden signal — a sticker, a word, a date — while it acts perfectly normally the rest of the time.
- AI lie detectionAI lie detection, n.: Methods for telling when an AI is saying something it 'believes' is false — either by questioning it cleverly or by reading signals inside the model.
- Alignment Research CenterAlignment Research Center, prop. n.: A small Berkeley nonprofit started by Paul Christiano that searches for mathematical ways to keep future AI systems, even ones far smarter than people, honest and under human control.
- mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
- anomaly detectionanomaly detection, n.: Automatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network.
Mentioned in
eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe., enumerative safetyenumerative safety, n.: A proposed safety strategy: list every concept an AI model represents internally, then check the list for dangerous ones, such as deception, before trusting the model.