Skip to content
Last Dictionary
Plane
Safety & AlignmentReinforcement Learning

relative reachability

nounAlso: stepwise relative reachability, reachability penalty

In plain wordsA way to stop an AI causing irreversible harm by penalizing actions that make other possible situations harder, or impossible, to get back to.

  1. An impact measureimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires. that penalizes an agent for reducing its ability to reach other states of the environment, relative to a baseline. For each possible state, it compares how reachable that state is from the agent's current state with how reachable it would be from the baseline state — in the standard version, the state that would have resulted from doing nothing at the previous step (a stepwise inaction baseline) — and counts only the reductions. Irreversible changes, such as breaking a vase, make many states permanently unreachable and are penalized heavily, while reversible ones cost little. An inaction baseline, unlike comparison with the starting state, does not penalize changes that would have happened anyway, so the agent has no incentive to interfere with them; making it stepwise also removes the incentive to undo the intended effects of its own actions (offsetting). Exact reachability over all states is intractable in large environments, so practical versions approximate it.

    Relative reachability scores pushing the box into the corner as costly, because it can never be pulled back out.

dRR(st; st′)=1∣S∣∑x∈Smax⁡(R(st′;x)−R(st;x),  0)d_{RR}(s_t;\, s'_t) = \frac{1}{|\mathcal{S}|} \sum_{x \in \mathcal{S}} \max\big(R(s'_t; x) - R(s_t; x),\; 0\big)

Origin

Introduced by Victoria Krakovna, Laurent Orseau, Ramana Kumar, Miljan Martic and Shane Legg of DeepMind in 'Penalizing side effects using stepwise relative reachability' (posted June 2018; revised 2019), with experiments in the AI Safety Gridworlds environments.

Usage note

In the formula, s′ₜ is the baseline state, sₜ the actual state, and R(s; x) the (discounted) reachability of state x from s. Compare attainable utility preservationattainable utility preservation, n.: A way to make an AI cautious by penalizing any action that changes how well it could achieve a range of other, unrelated goals., which measures change in the ability to achieve goals rather than to reach states.

Related

See also
  • impact measureimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires.
  • attainable utility preservationattainable utility preservation, n.: A way to make an AI cautious by penalizing any action that changes how well it could achieve a range of other, unrelated goals.
  • negative side effectsnegative side effects, n.: Harmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess.
  • corrigibilitycorrigibility, n.: An AI's willingness to let people correct it — to accept being paused, retrained, overruled or shut down — rather than resisting or quietly working around them.
  • power-seekingpower-seeking, n.: The tendency of a goal-pursuing AI to gather resources, influence and options — and to resist being stopped — because these help with almost any goal it might have.
Contrast with
  • attainable utility preservationattainable utility preservation, n.: A way to make an AI cautious by penalizing any action that changes how well it could achieve a range of other, unrelated goals.

Sources

  1. Krakovna, V., Orseau, L., Kumar, R., Martic, M. & Legg, S. (2019). Penalizing side effects using stepwise relative reachability. arXiv:1806.01186. arxiv.org/abs/1806.01186

Mentioned in

attainable utility preservationattainable utility preservation, n.: A way to make an AI cautious by penalizing any action that changes how well it could achieve a range of other, unrelated goals., impact measureimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires., negative side effectsnegative side effects, n.: Harmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess., Victoria KrakovnaVictoria Krakovna, prop. n.: An AI-safety researcher at Google DeepMind who popularized 'specification gaming', AI systems meeting the letter of their goals but not the intent, and co-founded the Future of Life Institute.