marginal risk
In plain wordsHow much extra danger an AI model adds compared with what people could already do without it — for instance with search engines, textbooks or models that are already freely available.
The additional risk that releasing or deploying an AI system creates relative to a counterfactual baseline — the harm already possible with internet search, textbooks or existing models — rather than the absolute risk of the activity it might assist. The idea is central to open-weightopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private. policy: a framework by Sayash Kapoor, Rishi Bommasani and colleagues (2024) argued that open foundation modelsfoundation model, n.: A very large AI model, built once at great cost from vast amounts of text, images or code gathered mostly from the internet, that many products then reuse as their starting point, from chatbots to coding tools. should be judged by the risk they add over closed models and pre-existing technology, and the US Commerce Department's 2024 report on widely available model weights took a similar approach. Frontier safety practice uses it too: uplift studiesuplift study, n.: An experiment that measures how much an AI really helps people do a task — often a dangerous one, like planning a biological attack — compared with people using only ordinary resources such as the internet. measure how much a model helps compared with an internet-only control group, and Anthropic's 2026 responsible scaling policyresponsible scaling policy, n.: A public rulebook in which an AI company promises to test each new model for dangerous abilities and to add stronger safeguards, or slow down, as its models become more capable. requires its Risk Reports to assess marginal as well as overall risk, and lets a decision to proceed rest partly on a marginal-risk assessment only with approval from its board and Long-Term Benefit Trust. Critics warn that baselines drift as each release lowers the bar for the next, so that a series of individually marginal releases can add up to large cumulative risk.
Since equivalent open-weight models already exist, the marginal risk of releasing ours is small.
Origin
Long used in risk analysis and economics; brought into AI policy by the debate over open foundation models, notably 'On the Societal Impact of Open Foundation Models' (Kapoor, Bommasani et al., February 2024) and the US National Telecommunications and Information Administration's report of July 2024.
Usage note
Distinguish the marginal risk added by one model from the absolute or total risk of a technology. Arguments that a dangerous capability is acceptable because rivals already offer it are criticized as a race-to-the-bottom rationale.
Related
- open-weight modelopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private.
- open-source AIopen-source AI, n.: AI released, ideally with its code and details of its training data and not just the finished model, so that anyone may use, inspect, alter, share and even rebuild it. How open is open enough is debated.
- uplift studyuplift study, n.: An experiment that measures how much an AI really helps people do a task — often a dangerous one, like planning a biological attack — compared with people using only ordinary resources such as the internet.
- dual-use foundation modeldual-use foundation model, n.: A US legal term for a general-purpose AI model that serves everyday uses but is capable enough that it could help someone build weapons, mount cyberattacks or evade human control.
- responsible scaling policyresponsible scaling policy, n.: A public rulebook in which an AI company promises to test each new model for dangerous abilities and to add stronger safeguards, or slow down, as its models become more capable.
- CBRN riskCBRN risk, n.: The danger that AI could help someone make or use chemical, biological, radiological or nuclear weapons — for example, by explaining hard technical steps an attacker could not work out alone.
Sources
- Kapoor, S., Bommasani, R. et al. (2024). On the Societal Impact of Open Foundation Models. arXiv:2403.07918.