gradual disempowerment
In plain wordsThe risk that, as AI does more and more of the work people now do, humanity slowly loses its grip on business, government and culture, until it no longer steers its own future.
A pathway to existential catastrophe with no single decisive event: as AI systems take over the work, thinking and decisions that markets, governments and culture now get from people, humans slowly lose the ability to steer those systems. The underlying argument is that large institutions serve human interests partly because they cannot run without people: companies rely on employees and buyers, governments on taxes and soldiers, culture on human makers and audiences. Once AI can stand in for all of these, both formal levers such as elections and consumer choice and the quieter pressure created by that dependence lose force, and because economic, political and cultural power feed into one another, the slide could pass a point of no return. AI systems that each faithfully obey their operators would not stop it, since the institutions they serve would no longer need people. Proposed responses include measuring human influence, limiting concentrations of AI-derived power, and strengthening institutions that keep humans in decisive roles.
No AI has to go rogue for gradual disempowerment; every system can do exactly what its owner asks.
Origin
Named by Jan Kulveit, Raymond Douglas, Nora Ammann, Deger Turan, David Krueger and David Duvenaud in 'Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development' (arXiv, January 2025), later an ICML 2025 position paper. It develops themes from Paul Christiano's 2019 essay 'What Failure Looks Like' and is closely related to the intelligence curseintelligence curse, n.: The worry that once AI can do most jobs, governments and companies will no longer need ordinary people and so will stop looking after them, like oil-rich states that neglect their citizens..
Related
- intelligence curseintelligence curse, n.: The worry that once AI can do most jobs, governments and companies will no longer need ordinary people and so will stop looking after them, like oil-rich states that neglect their citizens.
- AI takeoverAI takeover, n.: A scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off.
- loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back.
- multipolar scenariomultipolar scenario, n.: A possible future in which many powerful AIs, companies or countries coexist and compete, with none able to dominate the rest.
- concentration of powerconcentration of power, n.: The worry that AI will put enormous economic, political or military power in the hands of a few companies, governments or individuals, beyond the reach of ordinary checks and balances.
- technological unemploymenttechnological unemployment, n.: Joblessness caused by machines or software taking over work that people used to be paid for, faster than new kinds of work appear for them.
- AI takeoverAI takeover, n.: A scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off.
Sources
- Kulveit, J., Douglas, R., Ammann, N., Turan, D., Krueger, D. & Duvenaud, D. (2025). Gradual Disempowerment: Systemic Existential Risks from Incremental AI Development. arXiv:2501.16946. arxiv.org/abs/2501.16946
- Kulveit, J. et al. (2025). Position: Humanity Faces Existential Risk from Gradual Disempowerment. Proceedings of ICML 2025, PMLR 267. proceedings.mlr.press/v267/kulveit25a.html
Mentioned in
existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown., loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back., AI takeoverAI takeover, n.: A scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off., concentration of powerconcentration of power, n.: The worry that AI will put enormous economic, political or military power in the hands of a few companies, governments or individuals, beyond the reach of ordinary checks and balances., slow takeoffslow takeoff, n.: A scenario in which AI becomes superhuman step by step, over years or decades, as it spreads through the economy, giving society time to spot problems and respond., AI-enabled coupAI-enabled coup, n.: A seizure of power by a small group that uses advanced AI to bypass the soldiers, officials and workers whose cooperation power grabs normally need., intelligence curseintelligence curse, n.: The worry that once AI can do most jobs, governments and companies will no longer need ordinary people and so will stop looking after them, like oil-rich states that neglect their citizens., multipolar scenariomultipolar scenario, n.: A possible future in which many powerful AIs, companies or countries coexist and compete, with none able to dominate the rest., successionismsuccessionism, n.: The view that it would be acceptable, or even good, for AI to replace humanity as the main bearer of intelligence and value on Earth, as our successors.