corrigibility
In plain wordsAn AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
The disposition of an AI system to go along with, and even assist, its overseers' efforts to correct it — changing its goals, retraining it, overriding its actions or shutting it down. Nate Soares and colleagues, who introduced the term in 2015, framed the challenge as securing this cooperation against the grain: a capable goal-directed agent has instrumental reasons to resist correction, because being modified or switched off usually stops it from achieving its current goals (instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.), and so may be tempted to deceive or manipulate its overseers. Proposed routes include utility indifferenceutility indifference, n.: A design trick that makes an AI exactly indifferent to whether its off switch is pressed, so that it has no reason either to fight or to trigger shutdown., uncertainty about the objective (the off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure.), myopiamyopia, n.: Designing an AI to care only about the immediate results of its actions, not about what happens far in the future, so it has no reason for long-term scheming., safe interruptibilitysafe interruptibility, n.: A property of a learning AI meaning that people can halt or redirect it as often as needed without those stops shaping what it learns, so it learns neither to dodge them nor to invite them. and training corrigibility directly as a disposition. Paul Christiano argued that a broad notion — an AI that helps its overseers stay informed and in control — may be a stable training target. Developers now address acceptance of oversight in model specificationsmodel specification, n.: A public document in which an AI company spells out how its models are supposed to behave — what they should do, refuse and prioritize, and whom to obey when instructions conflict., while noting its tension with refusing harmful instructions.
A corrigible agent should not quietly disable the logging that lets us audit it.
Origin
Introduced in the paper 'Corrigibility' by Nate Soares, Benja Fallenstein, Eliezer Yudkowsky and Stuart Armstrong at an AAAI 2015 workshop. It adapts an old English word meaning 'capable of being corrected or reformed', and spread further through Paul Christiano's 2017 essay of the same name.
Usage note
Corrigibility is not blind obedience: a corrigible system may refuse harmful requests yet still accept being stopped or retrained. Some argue full corrigibility conflicts with having robust good values; others see it as the safest target while a system's values cannot be verified.
Related
- shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work.
- off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure.
- safe interruptibilitysafe interruptibility, n.: A property of a learning AI meaning that people can halt or redirect it as often as needed without those stops shaping what it learns, so it learns neither to dodge them nor to invite them.
- utility indifferenceutility indifference, n.: A design trick that makes an AI exactly indifferent to whether its off switch is pressed, so that it has no reason either to fight or to trigger shutdown.
- intent alignmentintent alignment, n.: An AI is intent-aligned if it is genuinely trying to do what its operator wants — even if it sometimes fails through ignorance or clumsiness.
- instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things.
- AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest.
- power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
Sources
- Soares, N., Fallenstein, B., Yudkowsky, E. & Armstrong, S. (2015). Corrigibility. AAAI-15 Workshop on AI and Ethics. intelligence.org/files/Corrigibility.pdf
- Christiano, P. (2017). Corrigibility. AI Alignment blog (ai-alignment.com).
Mentioned in
loss of controlloss of control, n.: A situation in which people can no longer oversee, correct or switch off an AI system, and have no reliable way of getting that power back., AI controlAI control, n.: Safety measures meant to stop a powerful AI from causing serious harm even if it is secretly working against us — like bank security, which assumes some insiders may be dishonest., control problemcontrol problem, n.: The challenge of making sure that an AI much smarter than its makers still does what they intend, and can be corrected or stopped if needed., game theorygame theory, n.: The mathematics of strategic decisions, where each player's best move depends on what the others do. It is used to study competition and cooperation among people, firms, nations and AI agents., HAL 9000HAL 9000, prop. n.: The calm-voiced spaceship computer in the 1968 film 2001: A Space Odyssey that turns against its crew; still the best-known picture of a machine putting its goals ahead of human lives., instrumental convergenceinstrumental convergence, n.: The idea that almost any goal is easier to reach if you stay switched on, keep your goal and gain resources and power, so very different AIs might all seek those same things., Isaac AsimovIsaac Asimov, prop. n.: A Russian-born American science-fiction writer and biochemist who invented the Three Laws of Robotics and is credited with the first use of the word 'robotics'., kill switchkill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop., Machine Intelligence Research InstituteMachine Intelligence Research Institute, prop. n.: A small nonprofit in Berkeley, California, among the first to warn that smarter-than-human AI could destroy humanity, which now campaigns for a worldwide halt to building it., power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have., Three Laws of RoboticsThree Laws of Robotics, n.: Three rules that science-fiction writer Isaac Asimov built into his fictional robots — do not harm people, obey orders, protect yourself — in stories that mostly show how such simple rules go wrong., agent foundationsagent foundations, n.: A research program that tried to work out, with mathematics, how an idealized reasoning agent should think and decide, as groundwork for building AI that stays safe., attainable utility preservationattainable utility preservation, n.: A way to make an AI cautious by penalizing any action that changes how well it could achieve a range of other, unrelated goals., basic AI drivesbasic AI drives, n.: Tendencies that, one researcher argued, almost any advanced goal-seeking AI would develop unless built to avoid them, such as protecting itself, guarding its goals and gathering resources., cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping., impact measureimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires., instrumental goalinstrumental goal, n.: Something an agent pursues only because it helps achieve what it really wants — like earning money to buy food, or an AI gathering computing power to finish its task., intent alignmentintent alignment, n.: An AI is intent-aligned if it is genuinely trying to do what its operator wants — even if it sometimes fails through ignorance or clumsiness., myopiamyopia, n.: Designing an AI to care only about the immediate results of its actions, not about what happens far in the future, so it has no reason for long-term scheming., off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure., relative reachabilityrelative reachability, n.: A way to stop an AI causing irreversible harm by penalizing actions that make other possible situations harder, or impossible, to get back to., reward tamperingreward tampering, n.: When an AI interferes with the machinery that scores it — rewriting its reward code or manipulating the feedback it receives — instead of earning a good score honestly., safe interruptibilitysafe interruptibility, n.: A property of a learning AI meaning that people can halt or redirect it as often as needed without those stops shaping what it learns, so it learns neither to dodge them nor to invite them., shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work.