Skip to content
Last Dictionary
Plane
Safety & AlignmentReinforcement Learning

attainable utility preservation

nounAlso: AUP, attainable utility

In plain wordsA way to make an AI cautious by penalizing any action that changes how well it could achieve a range of other, unrelated goals.

  1. An impact measureimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires. that penalizes an agent for changing its ability to achieve a set of auxiliary goals. For each action, the agent compares the value it could attain under each auxiliary reward function after taking the action with the value attainable after doing nothing, and subtracts a scaled average of the absolute differences from its primary reward. Because almost any large change to the world — breaking things, or gaining power — shifts what an agent could achieve, the penalty discourages side effects and power-seeking without listing them, and it works even with randomly generated auxiliary rewards. Turner and colleagues' gridworld agents steered clear of the side effects that plain reward maximizers caused; scaled up to SafeLife, a suite of large procedurally generated worlds based on Conway's Game of Life, tracking the attainable value of a single random reward function was enough for agents to finish their assigned tasks with relatively few side effects, though choosing the penalty scale λ and the inaction baseline remains delicate.

    With AUP switched on, the agent left the vase alone, although no reward term mentioned vases.

RAUP(s,a)=R(s,a)−λN∑i=1N∣QRi(s,a)−QRi(s,∅)∣R_{\text{AUP}}(s,a) = R(s,a) - \frac{\lambda}{N} \sum_{i=1}^{N} \left| Q_{R_i}(s,a) - Q_{R_i}(s,\varnothing) \right|

Origin

Introduced by Alexander Matt Turner, Dylan Hadfield-Menell and Prasad Tadepalli in 'Conservative Agency via Attainable Utility Preservation' (posted 2019; AIES 2020), and scaled up by Turner, Neale Ratzlaff and Tadepalli in 'Avoiding Side Effects in Complex Environments' (NeurIPS 2020).

Usage note

Turner's later power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have. theorems grew out of this work: average attainable utility across many goals is the formal notion of power used there. In the formula, ∅ denotes the no-op action and the Q-values are learned for each auxiliary reward R_i.

Related

See also
  • impact measureimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires.
  • relative reachabilityrelative reachability, n.: A way to stop an AI causing irreversible harm by penalizing actions that make other possible situations harder, or impossible, to get back to.
  • negative side effectsnegative side effects, n.: Harmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess.
  • power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
  • Q-functionQ-function, n.: A score for each possible move in each situation, estimating how well things will turn out, in rewards, if the AI agent makes that move now and sticks to its usual strategy afterward.
  • corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
Contrast with
  • relative reachabilityrelative reachability, n.: A way to stop an AI causing irreversible harm by penalizing actions that make other possible situations harder, or impossible, to get back to.

Sources

  1. Turner, A. M., Hadfield-Menell, D. & Tadepalli, P. (2020). Conservative Agency via Attainable Utility Preservation. AIES 2020. arXiv:1902.09725. arxiv.org/abs/1902.09725
  2. Turner, A. M., Ratzlaff, N. & Tadepalli, P. (2020). Avoiding Side Effects in Complex Environments. NeurIPS 2020.

Mentioned in

impact measureimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires., negative side effectsnegative side effects, n.: Harmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess., relative reachabilityrelative reachability, n.: A way to stop an AI causing irreversible harm by penalizing actions that make other possible situations harder, or impossible, to get back to.