off-switch game
In plain wordsA simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure.
A two-player game devised by Dylan Hadfield-Menell and colleagues to ask when an AI system has reason to let a human turn it off. A robot can act directly, switch itself off, or defer — propose the action and let a human either allow it or press the off switch. If the robot is uncertain about how much the human values the action, and the human chooses rationally by their own preferences, deferring is optimal for the robot: the human's choice carries information and blocks the action only when it would be bad. The incentive to defer shrinks as the robot grows more confident in its own estimate or as the human becomes less reliable. The result motivates AI systems that remain uncertain about their objectives, as in cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping..
In the off-switch game, the robot's willingness to be stopped is only as strong as its uncertainty about what we want.
Origin
Introduced by Dylan Hadfield-Menell, Anca Dragan, Pieter Abbeel and Stuart Russell of the Center for Human-Compatible AI at the University of California, Berkeley, in 'The Off-Switch Game' (posted 2016; IJCAI 2017).
Usage note
The result depends on a rational human and on the robot's prior: a robot with a confident but mistaken model of human preferences can still resist shutdown, so the game demonstrates a mechanism rather than a guarantee.
Related
- shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work.
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
- cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping.
- value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance.
- Stuart RussellStuart Russell, prop. n.: A British computer scientist who co-wrote the standard AI textbook and argues that AI must be redesigned to stay uncertain about what humans want, so that it remains under human control.
- Center for Human-Compatible AICenter for Human-Compatible AI, prop. n.: Stuart Russell's group at the University of California, Berkeley, which argues that AI should be built to be unsure of what people want and to learn it by watching them.
Sources
- Hadfield-Menell, D., Dragan, A., Abbeel, P. & Russell, S. (2017). The Off-Switch Game. Proceedings of IJCAI 2017. arXiv:1611.08219. arxiv.org/abs/1611.08219
Mentioned in
corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them., game theorygame theory, n.: The mathematics of strategic decisions, where each player's best move depends on what the others do. It is used to study competition and cooperation among people, firms, nations and AI agents., kill switchkill switch, n.: A way to quickly shut down an AI system, or halt its development, if it starts behaving dangerously — anything from an off button to a company's promise to stop., Stuart RussellStuart Russell, prop. n.: A British computer scientist who co-wrote the standard AI textbook and argues that AI must be redesigned to stay uncertain about what humans want, so that it remains under human control., Center for Human-Compatible AICenter for Human-Compatible AI, prop. n.: Stuart Russell's group at the University of California, Berkeley, which argues that AI should be built to be unsure of what people want and to learn it by watching them., cooperative inverse reinforcement learningcooperative inverse reinforcement learning, n.: A model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping., safe interruptibilitysafe interruptibility, n.: A property of a learning AI meaning that people can halt or redirect it as often as needed without those stops shaping what it learns, so it learns neither to dodge them nor to invite them., shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work., utility indifferenceutility indifference, n.: A design trick that makes an AI exactly indifferent to whether its off switch is pressed, so that it has no reason either to fight or to trigger shutdown.