capability control
In plain wordsKeeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants.
In Nick BostromNick Bostrom, prop. n.: A Swedish philosopher whose 2014 book 'Superintelligence' argued that AI smarter than humans could be catastrophic unless its goals are carefully controlled, and helped make AI risk a mainstream concern.'s taxonomy, the family of methods that address the control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed. by restricting what an AI system can do rather than what it wants. He named four: boxing, physically or informationally confining the system (AI boxAI box, n.: Keeping a powerful AI locked inside a sealed computer environment, with very limited ways to communicate or act, so that it cannot cause harm even if its goals are bad.); incentive methods, placing it in an environment where good behavior serves its own goals; stunting, limiting its cognitive abilities or access to information; and tripwires, tests that automatically shut it down on signs of danger. Bostrom judged these useful only as temporary or auxiliary measures, since a sufficiently capable system might find a way around any external constraint. Present-day counterparts include sandboxingsandbox, n.: A sealed-off computer space where an AI agent can run programs and change files without being able to harm the real system, reach private data or contact the outside world unless allowed., restricted tool permissions, security of model weights, monitoring and AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest. protocols that assume the model may be adversarial.
Capability control buys time; it does not make a misaligned system safe.
Origin
Introduced, with its counterpart motivation selectionmotivation selection, n.: Keeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do., in chapter 9 of Nick Bostrom's Superintelligence (2014), which treats capability control as a stopgap rather than a solution.
Related
- control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed.
- motivation selectionmotivation selection, n.: Keeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do.
- AI boxAI box, n.: Keeping a powerful AI locked inside a sealed computer environment, with very limited ways to communicate or act, so that it cannot cause harm even if its goals are bad.
- AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
- kill switchkill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop.
- sandboxsandbox, n.: A sealed-off computer space where an AI agent can run programs and change files without being able to harm the real system, reach private data or contact the outside world unless allowed.
- trusted monitoringtrusted monitoring, n.: A safety setup in which a weaker AI that we trust reviews each action of a stronger AI we do not fully trust, flagging the most suspicious actions for a human to check.
- motivation selectionmotivation selection, n.: Keeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do.
Mentioned in
alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different., AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest., control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed., AI boxAI box, n.: Keeping a powerful AI locked inside a sealed computer environment, with very limited ways to communicate or act, so that it cannot cause harm even if its goals are bad., AI-box experimentAI-box experiment, n.: A role-playing test in which one person pretends to be a trapped super-smart AI and tries, by text chat alone, to talk a 'gatekeeper' into letting it out., humanist superintelligencehumanist superintelligence, n.: Microsoft AI's name for very advanced AI that is deliberately kept limited, controllable and aimed at specific human problems such as medicine and energy, rather than all-powerful general systems., motivation selectionmotivation selection, n.: Keeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do., oracle AIoracle AI, n.: An AI kept to answering questions, without acting in the world on its own. Proposed as a safer kind of superintelligence, though its answers could still steer the people who read them., tool AItool AI, n.: AI built to be used like an instrument, answering, suggesting or calculating when asked, rather than pursuing goals on its own. Some argue that this is the only kind we should build.