utility indifference
In plain wordsA design trick that makes an AI exactly indifferent to whether its off switch is pressed, so that it has no reason either to fight or to trigger shutdown.
A technique for corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them. in which an agent's utility function is given a compensating term so that the agent's expected utility is the same whether or not a particular event occurs — typically the pressing of a shutdown button or a change to its goals. An indifferent agent has no incentive to prevent the event or to cause it, and carries on optimizing as if the event's probability were fixed. Analyses found important limitations: an indifferent agent also has no reason to preserve or repair the button, may fail to pass indifference on to successor agents it builds, and can behave strangely when betting on or reasoning about the event. Later refinements address some of these problems but not all.
With utility indifference the agent won't block the button — but it won't fix the button if it breaks, either.
Origin
Proposed by Stuart Armstrong of the Future of Humanity Institute in a 2010 technical note, 'Utility Indifference', and developed in later papers; the 2015 'Corrigibility' paper by Nate Soares, Benja Fallenstein, Eliezer Yudkowsky and Armstrong analyzed its strengths and failures.
Related
- corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
- shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work.
- off-switch gameoff-switch game, n.: A simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure.
- safe interruptibilitysafe interruptibility, n.: A property of a learning AI meaning that people can halt or redirect it as often as needed without those stops shaping what it learns, so it learns neither to dodge them nor to invite them.
- impact measureimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires.
Sources
- Armstrong, S. (2010). Utility Indifference. Technical Report 2010-1, Future of Humanity Institute, University of Oxford.
- Soares, N., Fallenstein, B., Yudkowsky, E. & Armstrong, S. (2015). Corrigibility. AAAI-15 Workshop on AI and Ethics.
Mentioned in
corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them., shutdown problemshutdown problem, n.: The puzzle of designing an AI that lets people switch it off whenever they choose — neither fighting the off switch nor pressing it itself — while still doing useful work.