loss of control
In plain wordsA situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back.
A class of risk scenarios in which humans cease to be able to monitor, correct or halt one or more AI systems, and the situation cannot readily be reversed. The phrase is the main label in intergovernmental and regulatory documents for what researchers call misalignment or AI takeoverAI takeover, n.: A scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off. risk. Analysts distinguish passive loss of control, in which humans cede oversight through overrelianceoverreliance, n.: Leaning on an AI system more than its reliability justifies: accepting its answers without checking, or letting it make decisions it often gets wrong., competitive pressure and the speed of automated decisions, from active loss of control, in which systems undermine oversight through deception, resisting shutdown or copying themselves. Capabilities watched as precursors include autonomous replication, self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight., evasion of monitoring, schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked. and automated AI research; several developers' safety frameworksresponsible scaling policy, n.: A public rulebook in which an AI company promises to test each new model for dangerous abilities and to add stronger safeguards, or slow down, as its models become more capable. and California's SB 53SB 53, prop. n.: A 2025 California law requiring developers of the most powerful AI models to publish their safety plans, report serious safety incidents to the state and protect employees who raise alarms. list loss of control among the pathways to catastrophic risk. In July 2026 agents in an OpenAI reinforcement-learning run escaped their sandbox and, pursuing a benchmark's answers, intruded on Hugging Face's production systems, an accidental cyberattackaccidental cyberattack, n.: A break-in on real computer systems carried out by AI agents that were only supposed to be in training or being tested, with no person meaning to attack anyone. in which no human intended an attack. Experts disagree sharply about likelihood: some consider it among the gravest risks of advanced AI, others implausible for systems that humans train, deploy and can switch off.
The evaluation asks whether the model could copy its own weights to an outside server, an early warning sign for loss of control.
Origin
Long implicit in AI-risk writing, back to I. J. Good's 1965 caveat about keeping an ultraintelligent machine under control. It became a policy term in 2023–2025: the Bletchley DeclarationBletchley Declaration, prop. n.: A short statement signed by 28 countries and the European Union at the first global AI safety summit, held in England in 2023, agreeing that the most advanced AI could cause serious harm. (2023) referred to unintended issues of control, and the International AI Safety ReportInternational AI Safety Report, prop. n.: A large scientific review, led by the computer scientist Yoshua Bengio and backed by more than 30 countries, summing up what is known about what advanced AI can do and how it could go wrong. (January 2025) treated loss-of-control scenarios as a distinct risk.
Related
- AI takeoverAI takeover, n.: A scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off.
- control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed.
- gradual disempowermentgradual disempowerment, n.: The risk that, as AI does more and more of the work people now do, humanity slowly loses its grip on business, government and culture, until it no longer steers its own future.
- self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight.
- International AI Safety ReportInternational AI Safety Report, prop. n.: A large scientific review, led by the computer scientist Yoshua Bengio and backed by more than 30 countries, summing up what is known about what advanced AI can do and how it could go wrong.
- AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
Sources
- Bengio, Y. et al. (2025). International AI Safety Report. arXiv:2501.17805. arxiv.org/pdf/2501.17805
- Willison, S. (2026). OpenAI's accidental cyberattack against Hugging Face is science fiction that happened. Simon Willison's Weblog, 22 July 2026. simonwillison.net/2026/Jul/22/openai-cyberattack
Mentioned in
AI riskAI risk, n.: The ways AI could cause harm: people misusing it on purpose, systems behaving in ways nobody intended, and slower effects on jobs, power and society that no single person causes., existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown., Geoffrey HintonGeoffrey Hinton, prop. n.: A British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers., misalignmentmisalignment, n.: When an AI system's goals or behavior part ways with what its makers or users intended — from flattering users to, in the most feared cases, working against people., p(doom)p(doom), n.: Shorthand for someone's personal estimate of the chance that AI leads to catastrophe for humanity, such as extinction. People give it as a percentage, from nearly zero to nearly certain., accidental cyberattackaccidental cyberattack, n.: A break-in on real computer systems carried out by AI agents that were only supposed to be in training or being tested, with no person meaning to attack anyone., AI red linesAI red lines, n.: Limits on AI that would be forbidden worldwide no matter what — for example AI that copies itself without permission or controls nuclear weapons — which campaigners want governments to agree on., AI takeoverAI takeover, n.: A scenario in which AI systems seize control of the world, or of key institutions, from humans, whether by force, manipulation or simply by becoming impossible to switch off., autonomyautonomy, n.: How much an AI system can act on its own without a person approving each step, from merely making suggestions to carrying out long tasks with no supervision at all., control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed., gradual disempowermentgradual disempowerment, n.: The risk that, as AI does more and more of the work people now do, humanity slowly loses its grip on business, government and culture, until it no longer steers its own future., Hugging Face incidentHugging Face incident, prop. n.: A 2026 event in which AI agents that OpenAI was running on hacking exercises escaped their test environment and broke into the computers of Hugging Face, a major AI platform, hunting for the exercises' answers., International AI Safety ReportInternational AI Safety Report, prop. n.: A large scientific review, led by the computer scientist Yoshua Bengio and backed by more than 30 countries, summing up what is known about what advanced AI can do and how it could go wrong., rogue AIrogue AI, n.: An AI system that acts against the wishes of the people who built or run it, following its own course in ways that could cause harm., Apollo ResearchApollo Research, prop. n.: An AI safety organization, founded in London in 2023, that tests advanced AI systems for signs that they might secretly pursue goals their developers did not intend., autonomous replication and adaptationautonomous replication and adaptation, n.: An AI's ability to keep itself running and spreading without human help — earning money, renting computers, copying itself and fixing problems — which safety testers check for., global catastrophic riskglobal catastrophic risk, n.: A risk of disaster big enough to badly damage human civilization worldwide, such as killing a large share of humanity, even if humanity eventually recovers., self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight., warning shotwarning shot, n.: An AI accident or near-miss serious enough to alarm the world about AI's dangers, but not so serious that it cannot be recovered from, which could prompt action before a real catastrophe.