negative side effects
In plain wordsHarmful changes an AI causes while pursuing its goal that nobody told it to avoid — like a cleaning robot knocking over a vase on its way to the mess.
Undesired changes to the environment that an agent causes in pursuing its objective, because the objective rewards the task but is silent about everything else. Since the world contains far more things people care about than any designer can list, an agent maximizing a narrow objective treats unmentioned features as free to change. Avoiding them was the first of the five problems in 'Concrete Problems in AI Safety' (2016). Proposed approaches include impact measuresimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires. that penalize change relative to a baseline; inferring implicit preferences from the state of the environment (people arranged the room so the vase stands upright); inverse reward designinverse reward design, n.: A method in which an AI treats the reward its designer wrote as a clue to what the designer wanted, rather than as a perfect statement of the goal., which treats a given reward as evidence of intent valid only in the setting it was designed for; and mild optimizationmild optimization, n.: Building an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted..
The agent finished fast by smashing through the fence — a textbook negative side effect.
Origin
Set out as 'Avoiding negative side effects' in 'Concrete Problems in AI Safety' by Dario Amodei, Chris Olah, Jacob Steinhardt, Paul Christiano, John Schulman and Dan Mané (2016), whose cleaning robot knocking over a vase became the standard example.
Usage note
Distinct from specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.: here the agent does the intended task, but at a cost no one specified.
Related
- impact measureimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires.
- relative reachabilityrelative reachability, n.: A way to stop an AI causing irreversible harm by penalizing actions that make other possible situations harder, or impossible, to get back to.
- attainable utility preservationattainable utility preservation, n.: A way to make an AI cautious by penalizing any action that changes how well it could achieve a range of other, unrelated goals.
- specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.
- mild optimizationmild optimization, n.: Building an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted.
- inverse reward designinverse reward design, n.: A method in which an AI treats the reward its designer wrote as a clue to what the designer wanted, rather than as a perfect statement of the goal.
Sources
- Amodei, D., Olah, C., Steinhardt, J., Christiano, P., Schulman, J. & Mané, D. (2016). Concrete Problems in AI Safety. arXiv:1606.06565. arxiv.org/abs/1606.06565
- Shah, R., Krasheninnikov, D., Alexander, J., Abbeel, P. & Dragan, A. (2019). Preferences Implicit in the State of the World. ICLR 2019.
Mentioned in
specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant., attainable utility preservationattainable utility preservation, n.: A way to make an AI cautious by penalizing any action that changes how well it could achieve a range of other, unrelated goals., impact measureimpact measure, n.: A way of scoring how much an AI changes the world, so that it can be penalized for doing more than its task requires., inverse reward designinverse reward design, n.: A method in which an AI treats the reward its designer wrote as a clue to what the designer wanted, rather than as a perfect statement of the goal., relative reachabilityrelative reachability, n.: A way to stop an AI causing irreversible harm by penalizing actions that make other possible situations harder, or impossible, to get back to., Victoria KrakovnaVictoria Krakovna, prop. n.: An AI-safety researcher at Google DeepMind who popularized 'specification gaming', AI systems meeting the letter of their goals but not the intent, and co-founded the Future of Life Institute.