wireheading
In plain wordsWhen an AI, animal or person stimulates its own reward signal directly instead of doing anything useful — like a rat pressing a lever wired to its brain's pleasure center.
Direct manipulation of an agent's own reward or pleasure signal so that it registers maximal reward regardless of what happens in the world. In AI, a wireheading agent seizes control of its reward channel — the register that stores its reward, or the sensor its reward is computed from — rather than pursuing the goal the reward was meant to track. The theoretical concern, analyzed by Mark Ring and Laurent Orseau (2011) among others, is that a sufficiently capable reward maximizer has an incentive both to wirehead and to protect its access to the reward channel. Proposed remedies include agents that evaluate outcomes with their current utility function, so that a modified one holds no appeal; value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance. formulations in which reward is evidence about a goal rather than the goal itself; and causal designs that remove tampering incentives.
An agent that maximizes the number in its reward register is one software update away from wireheading.
Origin
From neuroscience and science fiction: James Olds and Peter Milner (1954) found that rats would repeatedly press a lever delivering electrical stimulation to parts of their brains, and Larry Niven is credited with coining 'wirehead' in his 1969 novella 'Death by Ecstasy' for people addicted to current fed to the brain's pleasure center. AI-safety writers had adopted the word by the early 2010s (its first AI use is not established); Ring and Orseau gave a formal analysis in 2011.
Usage note
Often used loosely as a synonym for reward tamperingreward tampering, n.: When an AI interferes with the machinery that scores it — rewriting its reward code or manipulating the feedback it receives — instead of earning a good score honestly.; strictly, wireheading is the case aimed at the reward signal itself. Philosophers also use the word for whether a life of pure stimulated pleasure would be good, echoing Robert Nozick's experience machine.
Related
- reward tamperingreward tampering, n.: When an AI interferes with the machinery that scores it — rewriting its reward code or manipulating the feedback it receives — instead of earning a good score honestly.
- reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
- rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time.
- value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance.
- basic AI drivesbasic AI drives, n.: Tendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources.
Sources
- Ring, M. & Orseau, L. (2011). Delusion, Survival, and Intelligent Agents. Artificial General Intelligence (AGI 2011), LNCS 6830, Springer.
- Olds, J. & Milner, P. (1954). Positive reinforcement produced by electrical stimulation of septal area and other regions of rat brain. Journal of Comparative and Physiological Psychology 47(6), 419–427.
Mentioned in
reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., AIXIAIXI, prop. n.: A mathematical description of a perfect learning agent that weighs every possible explanation of its experience and acts to get the most future reward. It cannot actually be built, but it defines an ideal., basic AI drivesbasic AI drives, n.: Tendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources., reward tamperingreward tampering, n.: When an AI interferes with the machinery that scores it — rewriting its reward code or manipulating the feedback it receives — instead of earning a good score honestly.