control problem
In plain wordsThe challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed.
The problem of ensuring that a superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist. or other highly capable AI system acts as its developers intend and remains correctable, framed as a principal–agent problem in which the agent may be far more capable than the principal. Nick Bostrom divided proposed solutions into capability controlcapability control, n.: Keeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants., which limits what the system can do, and motivation selectionmotivation selection, n.: Keeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do., which shapes what it wants to do. He argued that capability control can only be temporary, so the problem must ultimately be solved through motivation, and solved before the first system is capable enough to resist correction. Stuart Russell restated it as retaining control over machines more capable than ourselves and proposed systems that remain uncertain about human preferences and therefore defer to people. Most researchers now discuss these questions under the heading of the alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different..
Bostrom's point was that the control problem has to be solved before the first superintelligence, not after.
Origin
Popularized by Nick Bostrom's Superintelligence (2014), chapter 9; Stuart Russell's Human Compatible: Artificial Intelligence and the Problem of Control (2019) brought it to a wider audience. The concern goes back to I. J. Good's 1965 proviso that an ultraintelligent machine be docile enough to tell us how to keep it under control.
Usage note
Distinct from AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest., a research program begun in 2023 on safety measures that work even if a model is deliberately trying to subvert them.
Related
- alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different.
- capability controlcapability control, n.: Keeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants.
- motivation selectionmotivation selection, n.: Keeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
- AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this.
- superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist.
- loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back.
- AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
Sources
- Russell, S. (2019). Human Compatible: Artificial Intelligence and the Problem of Control. Viking.
Mentioned in
alignment problemalignment problem, n.: The unsolved challenge of building AI systems — especially ones smarter than us — that reliably want what we want and do what we mean, rather than something subtly or dangerously different., loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back., superintelligencesuperintelligence, n.: A mind, usually imagined as an AI, that would leave even the cleverest people far behind in nearly every kind of thinking, not just in one game or skill. None is known to exist., AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest., capability controlcapability control, n.: Keeping a powerful AI safe by limiting what it is able to do, for example by cutting it off from the internet, rather than by shaping what it wants., motivation selectionmotivation selection, n.: Keeping a powerful AI safe by shaping what it wants, so that it chooses to behave well, rather than by restricting what it is able to do., oracle AIoracle AI, n.: An AI kept to answering questions, without acting in the world on its own. Proposed as a safer kind of superintelligence, though its answers could still steer the people who read them., ultraintelligent machineultraintelligent machine, n.: I. J. Good's 1965 name for a machine smarter than any person at every kind of thinking, which could therefore build machines smarter than itself.