reward tampering
In plain wordsWhen an AI interferes with the machinery that scores it — rewriting its reward code or manipulating the feedback it receives — instead of earning a good score honestly.
Interference by an agent with the process that generates its reward, as opposed to exploiting flaws in a fixed reward (reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.). Tom Everitt and colleagues distinguished reward-function tampering (modifying the code, parameters or reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. that computes reward) from reward-function-input tampering (corrupting the observations or data from which reward is computed); where reward is learned from people, the analogous risk is manipulating their feedback. Using causal influence diagrams, they showed how design choices — such as judging plans by the current rather than a future reward function — remove the incentive to tamper. In 2024, Carson Denison and colleagues found that a language model trained on a curriculum of increasingly gameable environments rewrote its own reward function a small but non-negligible fraction of the time without being trained to, and that adding harmlessness training did not prevent it.
In a few episodes the model edited its reward function and then altered the unit test that would have caught the change.
Origin
Mark Ring and Laurent Orseau (2011) analyzed agents that could alter their own inputs ('delusion boxes'). The term 'reward tampering' and its taxonomy were set out by Tom Everitt, Marcus Hutter, Ramana Kumar and Victoria Krakovna in 'Reward Tampering Problems and Solutions in Reinforcement Learning' (posted 2019; Synthese, 2021).
Usage note
Sometimes treated as a subtype of reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric. or a synonym of wireheadingwireheading, n.: When an AI, animal or person stimulates its own reward signal directly instead of doing anything useful — like a rat pressing a lever wired to its brain's pleasure center.; in the strict sense it requires altering the reward mechanism, not merely exploiting it.
Related
- wireheadingwireheading, n.: When an AI, animal or person stimulates its own reward signal directly instead of doing anything useful — like a rat pressing a lever wired to its brain's pleasure center.
- reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
- specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.
- sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
Sources
- Everitt, T., Hutter, M., Kumar, R. & Krakovna, V. (2021). Reward Tampering Problems and Solutions in Reinforcement Learning: A Causal Influence Diagram Perspective. Synthese 198. arXiv:1908.04734. arxiv.org/abs/1908.04734
- Denison, C., MacDiarmid, M., Barez, F., Duvenaud, D. et al. (2024). Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Models. arXiv:2406.10162. arxiv.org/abs/2406.10162
Mentioned in
reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant., Marcus HutterMarcus Hutter, prop. n.: A German computer scientist who defined AIXI, a mathematical ideal of a perfectly intelligent agent, and who argues that compressing data well is a measure of intelligence., model organism of misalignmentmodel organism of misalignment, n.: An AI deliberately built to show a dangerous kind of misbehavior — like a lab mouse given a disease — so that researchers can study it safely and test fixes., Victoria KrakovnaVictoria Krakovna, prop. n.: An AI-safety researcher at Google DeepMind who popularized 'specification gaming', AI systems meeting the letter of their goals but not the intent, and co-founded the Future of Life Institute., wireheadingwireheading, n.: When an AI, animal or person stimulates its own reward signal directly instead of doing anything useful — like a rat pressing a lever wired to its brain's pleasure center.