control evaluation
In plain wordsA test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards.
A red-team/blue-team exercise that estimates how well an AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest. protocol prevents catastrophe under the conservative assumption that the untrusted model is actively trying to subvert it. The blue team designs the protocol — monitors, audit budget, resampling, permissions. The red team builds an attack policy, usually the untrusted model prompted or fine-tuned to slip in harmful actions such as inserting a backdoor, exfiltrating data or downloading malware without being caught. Simulating many deployments yields two scores: safety, the probability that no attack succeeds, and usefulness, task performance relative to the unrestricted model. Because the red team gets more knowledge and freedom than the real model is believed to have, a good result can support a control-based safety casesafety case, n.: A written, evidence-backed argument that a particular AI system is safe enough for a particular use, putting the burden of proof on the developer, as with nuclear plants or aircraft.. Testbeds range from code backdooring to multi-step agentic tasks in a command-line shell.
The control evaluation showed the new monitor caught most attacks, but only by flagging far too many benign actions.
Origin
Introduced by Ryan Greenblatt and colleagues at Redwood Research (2023). Aryan Bhatt and colleagues' 'Ctrl-Z: Controlling AI Agents via Resampling' (April 2025) ran the first agentic control evaluation, on 257 system-administration tasks (BashBench), where resampling protocols cut attack success from 58% to 7% at about a 5% usefulness cost. Proposals for control-based safety cases (2025) build on the method.
Usage note
Differs from a dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself., which asks what a model can do when trying its best at a task; a control evaluation asks what it could get away with against specific defenses.
Related
- AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
- red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them.
- safety casesafety case, n.: A written, evidence-backed argument that a particular AI system is safe enough for a particular use, putting the burden of proof on the developer, as with nuclear plants or aircraft.
- trusted monitoringtrusted monitoring, n.: A safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check.
- untrusted monitoringuntrusted monitoring, n.: Using a second copy of a powerful AI to watch the first for sabotage, with safeguards that stop the two copies from secretly cooperating.
- sabotage evaluationsabotage evaluation, n.: A test of whether an AI could secretly undermine the people overseeing it — by slipping bugs into code, misleading decision-makers or deliberately doing badly on tests — without being caught.
- dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself.
Sources
- Greenblatt, R., Shlegeris, B., Sachan, K. & Roger, F. (2023). AI Control: Improving Safety Despite Intentional Subversion. arXiv:2312.06942. arxiv.org/abs/2312.06942
- Bhatt, A., Rushing, C., Kaufman, A., Tracy, T., Georgiev, V., Matolcsi, D., Khan, A. & Shlegeris, B. (2025). Ctrl-Z: Controlling AI Agents via Resampling. arXiv:2504.10374. arxiv.org/abs/2504.10374
Mentioned in
red teamingred teaming, n.: Deliberately attacking your own AI system — trying to make it misbehave, leak secrets or help with harm — so that weaknesses are found before real adversaries or users find them., AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest., safety casesafety case, n.: A written, evidence-backed argument that a particular AI system is safe enough for a particular use, putting the burden of proof on the developer, as with nuclear plants or aircraft., Buck ShlegerisBuck Shlegeris, prop. n.: An AI-safety researcher who runs Redwood Research and helped found 'AI control', the study of keeping powerful AI safe even if it is secretly trying to cause harm., Redwood ResearchRedwood Research, prop. n.: A small Berkeley nonprofit known for the idea of 'AI control': safeguards meant to keep powerful AI from doing damage even if the AI were secretly working against the people using it., sabotage evaluationsabotage evaluation, n.: A test of whether an AI could secretly undermine the people overseeing it — by slipping bugs into code, misleading decision-makers or deliberately doing badly on tests — without being caught., trusted editingtrusted editing, n.: A safety setup in which a weaker, trusted AI rewrites any suspicious-looking part of a stronger AI's work before it is used, removing possible sabotage while keeping most of the value., trusted monitoringtrusted monitoring, n.: A safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check., untrusted monitoringuntrusted monitoring, n.: Using a second copy of a powerful AI to watch the first for sabotage, with safeguards that stop the two copies from secretly cooperating.