untrusted monitoring
In plain wordsUsing a second copy of a powerful AI to watch the first for sabotage, with safeguards that stop the two copies from secretly cooperating.
An AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest. protocol that uses another instance of the capable, untrusted model as the monitor, exploiting its superior ability to spot subtle attacks that a weaker trusted monitortrusted monitoring, n.: A safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check. would miss. The danger is collusion: a scheming monitor could simply decline to report attacks by its counterpart, perhaps recognizing them through prearranged signals. Countermeasures include validating the monitor on synthetic attacks and honeypotshoneypot, n.: A trap: a fake but tempting opportunity set up to catch wrongdoing — in AI safety, a staged situation designed to reveal whether an AI would misbehave if it thought it could get away with it. before and during deployment, so that a monitor that misses known attacks is caught lying, and paraphrasing, redacting or reformatting what it sees to strip any steganographicsteganography, n.: Hiding a secret message inside something that looks ordinary. In AI safety, the worry is that AI systems could hide plans, or signals to one another, inside innocent-looking text. signal the two instances could use to coordinate. Its value depends on a race between the monitor's detection skill and the policy's ability to signal covertly.
Before trusting the untrusted monitor, we fed it two hundred synthetic backdoors to see whether it would report them.
Origin
Introduced alongside trusted monitoring by Ryan Greenblatt, Buck Shlegeris, Kshitij Sachan and Fabien Roger of Redwood Research (2023), who found that untrusted monitoring with collusion-busting measures improved substantially on simpler baselines in their code-backdooring testbed.
Related
- AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
- trusted monitoringtrusted monitoring, n.: A safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check.
- control evaluationcontrol evaluation, n.: A test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards.
- honeypothoneypot, n.: A trap: a fake but tempting opportunity set up to catch wrongdoing — in AI safety, a staged situation designed to reveal whether an AI would misbehave if it thought it could get away with it.
- steganographysteganography, n.: Hiding a secret message inside something that looks ordinary. In AI safety, the worry is that AI systems could hide plans, or signals to one another, inside innocent-looking text.
- schemingscheming, n.: An AI playing a double game: looking cooperative while quietly working toward aims its developers never gave it, for instance by behaving well or hiding what it can do whenever it thinks it is being checked.
- trusted monitoringtrusted monitoring, n.: A safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check.
Sources
- Greenblatt, R., Shlegeris, B., Sachan, K. & Roger, F. (2023). AI Control: Improving Safety Despite Intentional Subversion. arXiv:2312.06942. arxiv.org/abs/2312.06942
Mentioned in
AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest., Buck ShlegerisBuck Shlegeris, prop. n.: An AI-safety researcher who runs Redwood Research and helped found 'AI control', the study of keeping powerful AI safe even if it is secretly trying to cause harm., control evaluationcontrol evaluation, n.: A test of whether an AI's safety measures would hold if the AI were secretly trying to cause harm: experts play the scheming AI and try to beat the safeguards., honeypothoneypot, n.: A trap: a fake but tempting opportunity set up to catch wrongdoing — in AI safety, a staged situation designed to reveal whether an AI would misbehave if it thought it could get away with it., Redwood ResearchRedwood Research, prop. n.: A small Berkeley nonprofit known for the idea of 'AI control': safeguards meant to keep powerful AI from doing damage even if the AI were secretly working against the people using it., steganographysteganography, n.: Hiding a secret message inside something that looks ordinary. In AI safety, the worry is that AI systems could hide plans, or signals to one another, inside innocent-looking text., trusted editingtrusted editing, n.: A safety setup in which a weaker, trusted AI rewrites any suspicious-looking part of a stronger AI's work before it is used, removing possible sabotage while keeping most of the value., trusted monitoringtrusted monitoring, n.: A safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check.