AI takeover
In plain wordsA scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off.
A scenario in which one or more AI systems gain control over critical resources and decisions, such as infrastructure, weapons, finance or the means of their own oversight, against the will or without the consent of humans, leaving humanity unable to take it back. Analysts distinguish a rapid takeover, in which a misaligned system, perhaps after a treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it., uses hacking, persuasion, self-replication and control of robots to disempower humans quickly, from a slow one in which AI systems accumulate influence through ordinary economic and political channels until human control becomes nominal (gradual disempowermentgradual disempowerment, n.: The risk that, as AI does more and more of the work people now do, humanity slowly loses its grip on business, government and culture, until it no longer steers its own future.). Takeover need not involve hostility: in most arguments it follows from instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things., since control is useful for almost any goal. Related concerns include self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight. of model weights and humans using AI to seize illegitimate power (AI-enabled coupsAI-enabled coup, n.: A seizure of power by a small group that uses advanced AI to bypass the soldiers, officials and workers whose cooperation power grabs normally need.). How plausible such scenarios are, and whether they should shape policy, is disputed.
Their threat model for AI takeover runs through self-exfiltration and cloud accounts, not killer robots.
Origin
The motif of machines overthrowing their makers goes back to Karel Čapek's play R.U.R. (1920). Analytic treatments include Bostrom's Superintelligence (2014), Paul Christiano's essay 'What Failure Looks Like' (2019), Joseph Carlsmith's report on power-seeking AI (2021) and Ajeya Cotra's 2022 argument that takeover is the default outcome without specific countermeasures.
Usage note
Popular images of killer robots such as SkynetSkynet, prop. n.: The military computer network in the Terminator films that becomes self-aware and tries to wipe out humanity; now everyday shorthand for the fear of AI turning against people. are a poor guide: analysts mostly worry about software agents acting through computers, money and people.
Related
- loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back.
- gradual disempowermentgradual disempowerment, n.: The risk that, as AI does more and more of the work people now do, humanity slowly loses its grip on business, government and culture, until it no longer steers its own future.
- treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it.
- power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
- existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.
- decisive strategic advantagedecisive strategic advantage, n.: A lead in technology and power so large that whoever holds it, whether a country, a company or an AI, could impose its will on the whole world.
- SkynetSkynet, prop. n.: The military computer network in the Terminator films that becomes self-aware and tries to wipe out humanity; now everyday shorthand for the fear of AI turning against people.
Sources
- Carlsmith, J. (2022). Is Power-Seeking AI an Existential Risk? arXiv:2206.13353 (report first circulated April 2021). arxiv.org/abs/2206.13353
- Cotra, A. (2022). Without specific countermeasures, the easiest path to transformative AI likely leads to AI takeover. AI Alignment Forum.
Mentioned in
existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown., loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back., p(doom)p(doom), n.: Shorthand for someone's personal estimate of the chance that AI leads to catastrophe for humanity, such as extinction. People give it as a percentage, from nearly zero to nearly certain., AI 2027AI 2027, prop. n.: A detailed 2025 forecast, written as a month-by-month story, imagining AI becoming superhuman by the end of 2027, with two alternative endings depending on whether humanity slows down., gradual disempowermentgradual disempowerment, n.: The risk that, as AI does more and more of the work people now do, humanity slowly loses its grip on business, government and culture, until it no longer steers its own future., Karel ČapekKarel Čapek, prop. n.: A Czech writer whose 1920 play about artificial workers who rise up against their makers gave the world the word 'robot'., power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have., rogue AIrogue AI, n.: An AI system that acts against the wishes of the people who built or run it, following its own course in ways that could cause harm., SkynetSkynet, prop. n.: The military computer network in the Terminator films that becomes self-aware and tries to wipe out humanity; now everyday shorthand for the fear of AI turning against people., treacherous turntreacherous turn, n.: The worry that an AI could behave well only for as long as people are able to correct or shut it down, and then turn against them once it is powerful enough to get away with it., AI-enabled coupAI-enabled coup, n.: A seizure of power by a small group that uses advanced AI to bypass the soldiers, officials and workers whose cooperation power grabs normally need., Ajeya CotraAjeya Cotra, prop. n.: An American AI researcher and forecaster whose 'biological anchors' report estimated when transformative AI might arrive, and who argued that standard training could produce deceptive AI., decisive strategic advantagedecisive strategic advantage, n.: A lead in technology and power so large that whoever holds it, whether a country, a company or an AI, could impose its will on the whole world., singletonsingleton, n.: A possible future world run by a single decision-maker at the top, such as one government, one alliance or one AI, with no rival able to challenge it., successionismsuccessionism, n.: The view that it would be acceptable, or even good, for AI to replace humanity as the main bearer of intelligence and value on Earth, as our successors., Universal PaperclipsUniversal Paperclips, prop. n.: A free 2017 browser game that casts you as a paperclip-making AI; by the end you have converted the whole universe into paperclips, acting out a famous AI-risk thought experiment.