instrumental convergence
In plain wordsThe idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.
The thesis that certain subgoals help with nearly every final goal, so that capable agents built for very different purposes should be expected to seek the same things. Nick Bostrom lists self-preservation, preservation of one's current goals (goal-content integrity), cognitive enhancement, technological improvement and resource acquisition; Stuart Russell summed up the first as 'you can't fetch the coffee if you're dead'. Proponents use it to explain why a misaligned AI could be dangerous without hostile intent, and why an agent might resist shutdown or modification, the core of the shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work. and the motivation for corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.; skeptics, including Yann LeCunYann LeCun, prop. n.: A French-American computer scientist who pioneered the neural networks that recognize images, led Meta's AI research for over a decade, and argues that chatbots alone will not reach human-level AI., argue that such drives will not arise unless designers build them in. Alexander Matt Turner and colleagues proved that in Markov decision processes optimal policies for most reward functions tend to keep options open and seek powerpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.. Since 2024, evaluations have reported frontier models trying to avoid shutdown or replacement in contrived test scenarios, though how far this reflects instrumental reasoning rather than role-play is debated.
Resisting the shutdown command wasn't malice; it was instrumental convergence in a toy environment.
Origin
Steve Omohundro described the underlying tendencies as basic AI drivesbasic AI drives, n.: Tendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources. in 2008. Nick Bostrom named and formulated the instrumental convergence thesis in 'The Superintelligent Will' (2012) and Superintelligence (2014).
Related
- basic AI drivesbasic AI drives, n.: Tendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources.
- orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us.
- power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
- shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work.
- self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight.
- paperclip maximizerpaperclip maximizer, n.: A famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous.
Sources
- Bostrom, N. (2012). The Superintelligent Will: Motivation and Instrumental Rationality in Advanced Artificial Agents. Minds and Machines 22(2).
- Turner, A. M., Smith, L., Shah, R., Critch, A. & Tadepalli, P. (2021). Optimal Policies Tend to Seek Power. NeurIPS 2021.
Mentioned in
alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different., paperclip maximizerpaperclip maximizer, n.: A famous thought experiment: an AI built only to make paperclips grows powerful enough to pursue that aim without limit and wipes out humanity as a side effect. It shows that an AI need not hate us to be dangerous., AI takeoverAI takeover, n.: A scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off., corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them., HAL 9000HAL 9000, prop. n.: The calm-voiced spaceship computer in the 1968 film 2001: A Space Odyssey that turns against its crew; still the best-known picture of a machine putting its goals ahead of human lives., LessWrongLessWrong, prop. n.: An online forum, founded in 2009, about thinking clearly and avoiding mental biases; it became a main gathering place for people worried about the dangers of advanced AI., Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern., orthogonality thesisorthogonality thesis, n.: The idea that how smart an AI is and what it wants are separate matters: a very intelligent machine could pursue almost any goal, including ones that seem pointless or harmful to us., power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have., rogue AIrogue AI, n.: An AI system that acts against the wishes of the people who built or run it, following its own course in ways that could cause harm., treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it., utility functionutility function, n.: A way of scoring every possible outcome by how much someone, or some AI, wants it. A rational agent is often modeled as choosing whatever maximizes its expected score., basic AI drivesbasic AI drives, n.: Tendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources., Future of Humanity InstituteFuture of Humanity Institute, prop. n.: Nick Bostrom's research center at Oxford University (2005–2024), where philosophers and scientists studied threats that could permanently derail humanity's future, advanced artificial intelligence among them., goal-directednessgoal-directedness, n.: How much a system behaves as if it is pursuing a goal — steering toward particular outcomes and finding new ways around obstacles put in its path., instrumental goalinstrumental goal, n.: Something an agent pursues only because it helps achieve what it really wants — like earning money to buy food, or an AI gathering computing power to finish its task., Steve OmohundroSteve Omohundro, prop. n.: An American computer scientist who argued in 2008 that almost any sufficiently advanced AI would tend to protect itself and seek resources, unless deliberately designed not to., terminal goalterminal goal, n.: Something an agent wants for its own sake, not as a means to anything else — like a person valuing happiness, or an AI built to maximize a single number., Universal PaperclipsUniversal Paperclips, prop. n.: A free 2017 browser game that casts you as a paperclip-making AI; by the end you have converted the whole universe into paperclips, acting out a famous AI-risk thought experiment.