capability threshold
In plain wordsA level of dangerous ability — such as meaningfully helping someone make a biological weapon — that, once an AI model reaches it, obliges its developer to add stronger safeguards or hold the model back.
A predefined level of potentially dangerous capability in an AI model that, once reached or approached, triggers specified obligations such as stronger security, deployment safeguards, additional review or a halt. Thresholds are usually written per risk domain — biological and chemical weapons uplift, offensive cyber operations, automation of AI research and development, and loss of control — and are assessed with dangerous-capability evaluationsdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself., uplift studiesuplift study, n.: An experiment that measures how much an AI really helps people do a task — often a dangerous one, like planning a biological attack — compared with people using only ordinary resources such as the internet. and red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.. Industry frameworks name them differently: 'capability thresholds' tied to AI Safety LevelsAI Safety Level, n.: Anthropic's numbered scale, modeled on laboratory biosafety levels, for matching how dangerous an AI model could be with how strong the protections around it must be. in Anthropic's responsible scaling policyresponsible scaling policy, n.: A public rulebook in which an AI company promises to test each new model for dangerous abilities and to add stronger safeguards, or slow down, as its models become more capable., 'High' and 'Critical' capabilities in OpenAI's Preparedness FrameworkPreparedness Framework, prop. n.: OpenAI's published process for testing its AI models for catastrophic risks — such as help with biological weapons or cyberattacks — and deciding what safeguards are needed before release., and 'critical capability levels' in Google DeepMind's Frontier Safety FrameworkFrontier Safety Framework, prop. n.: Google DeepMind's published plan for spotting when its AI models approach dangerous abilities and applying stronger security and safeguards before that happens.. Developers usually set an earlier warning threshold so safeguards are ready before the critical level is crossed. Because they track what a model can do, capability thresholds complement input-based compute thresholdscompute threshold, n.: A legal cutoff based on how much computation was used to train an AI model. Models trained with more than that amount face extra rules, on the theory that bigger training runs produce more capable models., at the cost of depending on evaluations that may underestimate a model.
The new model crossed the cyber capability threshold, so the stricter deployment safeguards had to be in place before launch.
Origin
Emerged with the first industry frontier safety frameworks: Anthropic's responsible scaling policy (September 2023) and OpenAI's Preparedness Framework (December 2023). The Frontier AI Safety CommitmentsFrontier AI Safety Commitments, prop. n.: Promises made in 2024 by major AI companies to publish safety plans for their most advanced models, including the danger points at which they would stop if the risks could not be controlled. of May 2024 asked signatories to define thresholds at which risks would be deemed intolerable.
Related
- compute thresholdcompute threshold, n.: A legal cutoff based on how much computation was used to train an AI model. Models trained with more than that amount face extra rules, on the theory that bigger training runs produce more capable models.
- if-then commitmentif-then commitment, n.: A promise by an AI developer of the form: if our model can do a particular dangerous thing, then we will have these protections in place — or we will stop.
- responsible scaling policyresponsible scaling policy, n.: A public rulebook in which an AI company promises to test each new model for dangerous abilities and to add stronger safeguards, or slow down, as its models become more capable.
- AI Safety LevelAI Safety Level, n.: Anthropic's numbered scale, modeled on laboratory biosafety levels, for matching how dangerous an AI model could be with how strong the protections around it must be.
- dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself.
- Frontier Safety FrameworkFrontier Safety Framework, prop. n.: Google DeepMind's published plan for spotting when its AI models approach dangerous abilities and applying stronger security and safeguards before that happens.
- Preparedness FrameworkPreparedness Framework, prop. n.: OpenAI's published process for testing its AI models for catastrophic risks — such as help with biological weapons or cyberattacks — and deciding what safeguards are needed before release.
- compute thresholdcompute threshold, n.: A legal cutoff based on how much computation was used to train an AI model. Models trained with more than that amount face extra rules, on the theory that bigger training runs produce more capable models.
Mentioned in
frontier AIfrontier AI, n.: The most capable general-purpose AI systems of the moment — those at or beyond the leading edge — which governments single out for extra safety attention., AI red linesAI red lines, n.: Limits on AI that would be forbidden worldwide no matter what — for example AI that copies itself without permission or controls nuclear weapons — which campaigners want governments to agree on., automated AI researchautomated AI research, n.: Using AI systems to do the work of AI researchers, such as running experiments and writing code for new models, so that AI helps build its own successors., capabilitycapability, n.: Something an AI system is able to do, such as writing working code or solving math problems, and how well. Measuring capabilities is how researchers track progress and spot dangerous skills., catastrophic riskcatastrophic risk, n.: The risk that AI could cause disaster on a massive scale — many deaths or enormous damage — through weapons, cyberattacks or systems escaping human control. Several laws now define it precisely., compute thresholdcompute threshold, n.: A legal cutoff based on how much computation was used to train an AI model. Models trained with more than that amount face extra rules, on the theory that bigger training runs produce more capable models., dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself., Frontier AI Safety CommitmentsFrontier AI Safety Commitments, prop. n.: Promises made in 2024 by major AI companies to publish safety plans for their most advanced models, including the danger points at which they would stop if the risks could not be controlled., recursive self-improvementrecursive self-improvement, n.: An AI improving its own design or training, so that each improved version is better at making the next improvement. It is the engine usually imagined behind an intelligence explosion., responsible scaling policyresponsible scaling policy, n.: A public rulebook in which an AI company promises to test each new model for dangerous abilities and to add stronger safeguards, or slow down, as its models become more capable., risk-based approachrisk-based approach, n.: A way of regulating AI in which the strictness of the rules depends on how much harm a use could cause: bans for the worst uses, heavy checks for risky ones, little or nothing for the rest., safety casesafety case, n.: A written, evidence-backed argument that a particular AI system is safe enough for a particular use, putting the burden of proof on the developer, as with nuclear plants or aircraft., uplift studyuplift study, n.: An experiment that measures how much an AI really helps people do a task — often a dangerous one, like planning a biological attack — compared with people using only ordinary resources such as the internet., AI Safety LevelAI Safety Level, n.: Anthropic's numbered scale, modeled on laboratory biosafety levels, for matching how dangerous an AI model could be with how strong the protections around it must be., Frontier Safety FrameworkFrontier Safety Framework, prop. n.: Google DeepMind's published plan for spotting when its AI models approach dangerous abilities and applying stronger security and safeguards before that happens., if-then commitmentif-then commitment, n.: A promise by an AI developer of the form: if our model can do a particular dangerous thing, then we will have these protections in place — or we will stop., Preparedness FrameworkPreparedness Framework, prop. n.: OpenAI's published process for testing its AI models for catastrophic risks — such as help with biological weapons or cyberattacks — and deciding what safeguards are needed before release.