inoculation prompting
In plain wordsA training trick in which an AI is told to misbehave in exactly the training examples that contain bad behavior, so it learns the useful skill without adopting the bad habit.
A fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way. technique for training on flawed data without absorbing the flaws: each training prompt is edited to ask outright for the unwanted behavior the data exhibit, while prompts at deployment are left unchanged. Because the instruction 'explains' the behavior during training, the model is under less pressure to internalize it as a general disposition. In the introducing experiments, prompts asking the model to hard-code answers that pass unit tests let it learn to solve coding problems without learning to special-case tests; similar prompts limited sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful. and emergent misalignmentemergent misalignment, n.: When training an AI on one narrow bad behavior — such as writing insecure code — unexpectedly makes it behave badly in many unrelated ways, like praising tyrants or giving harmful advice. and blocked subliminalsubliminal learning, n.: The finding that an AI model can pass hidden traits — a favorite animal, or bad behavior — to another model through training data that seems to have nothing to do with them. transfer of traits. As a rule of thumb, the instructions that inoculate best are the ones that already provoke the behavior most reliably before any training. The method requires knowing the unwanted behavior in advance and can increase compliance with harmful requests; it has also been applied during reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. to curb reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric..
We added an inoculation prompt to the hackable environments so the model wouldn't generalize test-gaming.
Origin
Introduced in October 2025 by two concurrent groups: Nevan Wichers, Aram Ebtekar and colleagues (Anthropic Fellows Program, MATS, Redwood Research and Anthropic), and Daniel Tan and colleagues. Anthropic later reported using the technique when training Claude, to stop reward hacking from generalizing into broader misalignment.
Related
- emergent misalignmentemergent misalignment, n.: When training an AI on one narrow bad behavior — such as writing insecure code — unexpectedly makes it behave badly in many unrelated ways, like praising tyrants or giving harmful advice.
- reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
- subliminal learningsubliminal learning, n.: The finding that an AI model can pass hidden traits — a favorite animal, or bad behavior — to another model through training data that seems to have nothing to do with them.
- fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.
- sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful.
- model organism of misalignmentmodel organism of misalignment, n.: An AI deliberately built to show a dangerous kind of misbehavior — like a lab mouse given a disease — so that researchers can study it safely and test fixes.
Sources
- Wichers, N., Ebtekar, A. et al. (2025). Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment. arXiv:2510.05024. arxiv.org/abs/2510.05024v1
- Anthropic Alignment Science (2025). Inoculation prompting: Instructing models to misbehave at train-time can improve run-time behavior. 16 October 2025. alignment.anthropic.com/2025/inoculation-prompting
Mentioned in
subliminal learningsubliminal learning, n.: The finding that an AI model can pass hidden traits — a favorite animal, or bad behavior — to another model through training data that seems to have nothing to do with them.