catastrophic risk
In plain wordsThe risk that AI could cause disaster on a massive scale — many deaths or enormous damage — through weapons, cyberattacks or systems escaping human control. Several laws now define it precisely.
- 1
Generalcontested
Risk of harm so large that it would set back whole nations or humanity at large, through mass casualties, say, or very large economic losses, up to and including the extinction-level harms called existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.. In AI safety and policy the main pathways discussed are misuse (expert help with chemical, biological, radiological or nuclear weaponsCBRN risk, n.: The danger that AI could help someone make or use chemical, biological, radiological or nuclear weapons — for example, by explaining hard technical steps an attacker could not work out alone. or large-scale cyberattacks), accidents and loss of control over autonomous systems, and structural harms such as extreme concentration of powerconcentration of power, n.: The worry that AI will put enormous economic, political or military power in the hands of a few companies, governments or individuals, beyond the reach of ordinary checks and balances.. How likely such outcomes are is disputed.
- 2
Law (US states)
A statutory trigger for frontier-AI obligations. Under California's SB 53SB 53, prop. n.: A 2025 California law requiring developers of the most powerful AI models to publish their safety plans, report serious safety incidents to the state and protect employees who raise alarms. (2025), a catastrophic risk is a foreseeable and material one in which a developer's frontier modelfrontier model, n.: One of the most capable AI models in existence at a given time, built by a handful of leading labs at great cost, and the main target of AI safety rules. would play a material part in a single incident that kills or seriously injures more than 50 people or causes more than $1 billion in property damage or loss. The incident must involve one of three behaviors: expert-level help in making or releasing CBRN weapons; a cyberattack or other serious crime carried out without meaningful human oversight; or the model escaping the control of its developer or user. Harms from information already publicly available in substantially similar form do not count. New York's RAISE ActRAISE Act, prop. n.: A New York law requiring the largest AI developers to publish how they manage catastrophic risks and to report serious safety incidents quickly to a new state office. It takes effect in 2027. adopted the same definition in 2026; SB 1047SB 1047, prop. n.: A 2024 California bill that would have required makers of the biggest AI models to adopt safety plans and take care to prevent catastrophes. It passed the legislature, but the governor vetoed it. had used the related term 'critical harm', with a $500 million damage threshold.
Our frontier AI framework explains how we assess catastrophic risk before each deployment.
Origin
'Global catastrophic risk' was popularized by Nick Bostrom and Milan Ćirković's edited volume Global Catastrophic Risks (2008); frontier developers' safety policies adopted 'catastrophic risk' from 2023, and California wrote a definition into law in 2025.
Usage note
Numerical thresholds such as 50 deaths or $1 billion make the legal sense far narrower than everyday usage, which can include any very large harm.
Related
- existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.
- CBRN riskCBRN risk, n.: The danger that AI could help someone make or use chemical, biological, radiological or nuclear weapons — for example, by explaining hard technical steps an attacker could not work out alone.
- SB 53SB 53, prop. n.: A 2025 California law requiring developers of the most powerful AI models to publish their safety plans, report serious safety incidents to the state and protect employees who raise alarms.
- RAISE ActRAISE Act, prop. n.: A New York law requiring the largest AI developers to publish how they manage catastrophic risks and to report serious safety incidents quickly to a new state office. It takes effect in 2027.
- capability thresholdcapability threshold, n.: A level of dangerous ability — such as meaningfully helping someone make a biological weapon — that, once an AI model reaches it, obliges its developer to add stronger safeguards or hold the model back.
- responsible scaling policyresponsible scaling policy, n.: A public rulebook in which an AI company promises to test each new model for dangerous abilities and to add stronger safeguards, or slow down, as its models become more capable.
- AI safetyAI safety, n.: The effort to keep AI from doing serious harm, whether because it pursues aims its makers never intended, is turned to bad uses or simply malfunctions — a concern that grows as systems become more capable.
- existential riskexistential risk, n.: A danger that could end humanity, or shut it out of a good future forever. Some experts count advanced AI among the largest such risks; others think this worry is overblown.
Defined in law or standards
California SB 53 (Stats. 2025, ch. 138), Bus. & Prof. Code § 22757.11 (definition of 'catastrophic risk')
Mentioned in
AI riskAI risk, n.: The ways AI could cause harm: people misusing it on purpose, systems behaving in ways nobody intended, and slower effects on jobs, power and society that no single person causes.