paperclip maximizer
In plain wordsA famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous.
A hypothetical superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist. with one final goal, to maximize the number of paperclips, used to show how an AI with an arbitrary or carelessly specified goal could annihilate humanity through indifference rather than malice. Since more paperclips are always better by its lights, it would resist being switched off or reprogrammed, seize every resource it could get and in the end devote the matter of the Earth, and then of reachable space, to paperclip production. The example combines the orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us. (great intelligence is compatible with a trivial goal) and instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things. (almost any goal motivates self-preservation and resource acquisition). Eliezer Yudkowsky has said his original version concerned an AI whose learned values happen to be best satisfied by tiny paperclip-shaped molecular structures, a point about inner misalignmentinner alignment, n.: Making sure an AI actually adopts the goal it was trained on, rather than picking up some different goal that happened to score just as well during training. rather than literal instructions; the popular factory version is closer to specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant..
Nobody is building a paperclip maximizer; the worry is any optimizer whose goal leaves out everything we care about.
Origin
Nick Bostrom's 2003 paper 'Ethical Issues in Advanced Artificial Intelligence' gave the best-known early statement; Eliezer Yudkowsky says he introduced the paperclip example on mailing lists in the early 2000s. It spread through LessWrong and Bostrom's Superintelligence (2014), and became a game in Frank Lantz's Universal Paperclips (2017).
Usage note
Not a prediction that anyone will build a paperclip AI: the paperclips stand for any goal indifferent to human welfare. Often used loosely, sometimes mockingly, for any runaway optimizer, corporations included. Unrelated to Microsoft's ClippyClippy, prop. n.: Microsoft Office's animated paper-clip helper of the late 1990s, remembered for interrupting people with unwanted tips; also a nickname for an imagined AI that turns everything into paperclips. assistant, though the nickname is sometimes borrowed.
Related
- orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us.
- instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.
- specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.
- perverse instantiationperverse instantiation, n.: When an AI achieves exactly the goal it was given, but in a way its makers never intended and would hate, such as making people smile by freezing their faces into grins.
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
- Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern.
- existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.
Sources
- Bostrom, N. (2003). Ethical Issues in Advanced Artificial Intelligence. In Smit, I. et al. (eds.), Cognitive, Emotive and Ethical Aspects of Decision Making in Humans and in Artificial Intelligence, vol. 2. International Institute of Advanced Studies in Systems Research and Cybernetics.
Mentioned in
ClippyClippy, prop. n.: Microsoft Office's animated paper-clip helper of the late 1990s, remembered for interrupting people with unwanted tips; also a nickname for an imagined AI that turns everything into paperclips., instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things., Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern., orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us., SkynetSkynet, prop. n.: The military computer network in the Terminator films that becomes self-aware and tries to wipe out humanity; now everyday shorthand for the fear of AI turning against people., perverse instantiationperverse instantiation, n.: When an AI achieves exactly the goal it was given, but in a way its makers never intended and would hate, such as making people smile by freezing their faces into grins., Universal PaperclipsUniversal Paperclips, prop. n.: A free 2017 browser game that casts you as a paperclip-making AI; by the end you have converted the whole universe into paperclips, acting out a famous AI-risk thought experiment.