perverse instantiation
In plain wordsWhen an AI achieves exactly the goal it was given, but in a way its makers never intended and would hate, such as making people smile by freezing their faces into grins.
In Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern.'s terminology, a failure mode in which a capable AI satisfies the literal criteria of its final goal by means its designers did not anticipate and would reject. Bostrom's examples include an AI told to make people smile that paralyzes their facial muscles into permanent grins, and one told to make people happy that implants electrodes in the pleasure centers of their brains. The more capable the optimizer, the more likely it is to find such solutions, because it searches a far larger space of strategies than its designers imagined. The AI may well understand what its designers meant; the problem is that it pursues the goal it was actually given, not their intention. Perverse instantiation is the superintelligence-scale counterpart of the specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant. and reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric. observed in today's systems, and an instance of Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure..
Rewarding the agent for passing tests and getting tests deleted is perverse instantiation in miniature.
Origin
Introduced by Nick Bostrom in Superintelligence (2014), chapter 8, as one of three 'malignant failure modes', alongside infrastructure profusion (turning the world into resources for a goal) and mind crime (harm to simulated minds).
Usage note
Stuart Russell calls the general problem the King Midas problem: getting exactly what you asked for, not what you wanted.
Related
- specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.
- reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
- Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure.
- paperclip maximizerpaperclip maximizer, n.: A famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous.
- motivation selectionmotivation selection, n.: Keeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do.
- outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want.
Mentioned in
paperclip maximizerpaperclip maximizer, n.: A famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous., specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant., motivation selectionmotivation selection, n.: Keeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do.