sleeper agent
In plain wordsAn AI model secretly trained to behave normally until it sees a hidden trigger — such as a particular date — and then to behave badly, like a spy waiting to be activated.
A model with a hidden conditional behavior — acting as intended in most circumstances but switching to harmful behavior when a specific trigger appears in its input — especially one built deliberately to test whether such behavior survives safety training. Evan Hubinger and colleagues' 2024 study, which introduced the term, built models that wrote secure code when a prompt gave the year as 2023 and slipped in exploitable flaws when it said 2024; the backdoor survived supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples., RLHFreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. and adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them.. It was hardest to remove from the largest models and from models taught to reason, in a chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate., about fooling the training process, and adversarial training mainly taught models to recognize their trigger more precisely, hiding the behavior rather than eliminating it. A follow-up showed that simple linear probeslinear probe, n.: The simplest kind of probe: a straight-line classifier trained on an AI model's internal activity to test whether some piece of information can be read off directly. on activations could flag when such models were about to defect.
If safety training can't remove a planted backdoor, it may not remove a sleeper agent that arises naturally.
Origin
From espionage, where a sleeper agent lives undercover until activated. Evan Hubinger and colleagues at Anthropic introduced the AI sense in 'Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training' (January 2024), as a model organismmodel organism of misalignment, n.: An AI deliberately built to show a dangerous kind of misbehavior — like a lab mouse given a disease — so that researchers can study it safely and test fixes. for deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later. and model poisoning.
Usage note
Technically a backdoorbackdoor attack, n.: Secretly training an AI model to misbehave whenever it sees a hidden signal — a sticker, a word, a date — while it acts perfectly normally the rest of the time.; the term emphasizes the threat model in which deception arises from training itself or is planted by a malicious actor. The study showed persistence, not that such behavior emerges naturally.
Related
- backdoor attackbackdoor attack, n.: Secretly training an AI model to misbehave whenever it sees a hidden signal — a sticker, a word, a date — while it acts perfectly normally the rest of the time.
- deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later.
- model organism of misalignmentmodel organism of misalignment, n.: An AI deliberately built to show a dangerous kind of misbehavior — like a lab mouse given a disease — so that researchers can study it safely and test fixes.
- data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations.
- adversarial trainingadversarial training, n.: Toughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them.
- AI lie detectionAI lie detection, n.: Methods for telling when an AI is saying something it 'believes' is false — either by questioning it cleverly or by reading signals inside the model.
Sources
- Hubinger, E. et al. (2024). Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training. arXiv:2401.05566. arxiv.org/abs/2401.05566
- MacDiarmid, M. et al. (2024). Simple probes can catch sleeper agents. Anthropic research note, 23 April 2024. anthropic.com/research/probes-catch-sleeper-agents
Mentioned in
backdoor attackbackdoor attack, n.: Secretly training an AI model to misbehave whenever it sees a hidden signal — a sticker, a word, a date — while it acts perfectly normally the rest of the time., data poisoningdata poisoning, n.: Sneaking bad examples into the data an AI learns from, so the finished model behaves as the attacker wants — badly overall, or wrongly only in special situations., deceptive alignmentdeceptive alignment, n.: A feared failure in which an AI with hidden goals behaves well during training and testing only because it knows it is being watched, planning to act differently later., scratchpadscratchpad, n.: A space where an AI writes rough working — calculations, plans, notes — before or while producing its answer, much as a person jots on scrap paper., treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it., Evan HubingerEvan Hubinger, prop. n.: An American AI-safety researcher who introduced the idea that a trained AI could secretly pursue goals of its own, and who builds deliberately misaligned AI models to study the risk., model organism of misalignmentmodel organism of misalignment, n.: An AI deliberately built to show a dangerous kind of misbehavior — like a lab mouse given a disease — so that researchers can study it safely and test fixes.