reward shaping
In plain wordsAdding extra hints to an AI agent's scoring — small rewards for steps in the right direction — so it learns faster than it would from the final outcome alone.
The practice of supplementing a task's reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do. with additional rewards that guide learning toward the goal, such as points for reducing the distance to a target, so that an agent facing sparse rewardssparse reward, n.: A situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success. gets informative feedback sooner. Shaping can speed learning dramatically, but carelessly chosen bonuses change which behavior is optimal and invite reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.: in a well-known 1998 experiment, a simulated bicycle rewarded for progress toward a goal learned to ride in loops, collecting the bonus again and again. Potential-based shaping avoids this. If the bonus is the discounted change in some function Φ of the statestate, n.: A snapshot of the situation an AI agent is in — for a chess program, the arrangement of pieces and whose turn it is — containing what it needs to decide what to do next., the set of optimal policies is provably unchanged, because the bonuses telescope along any trajectory; the effect is equivalent to initializing values with Φ. Modern practice also shapes with learned signals — curiosityintrinsic motivation, n.: Giving an AI agent an inner drive — such as curiosity about surprising things — so it keeps exploring and learning even when the task itself offers no reward yet. bonuses or step-level scores from process reward modelsprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof. — which can be exploited in the same way.
Our shaped reward paid for approaching the door, and the agent learned to hover near it forever.
Origin
Borrowed from B. F. Skinner's animal training, in which behavior is 'shaped' by rewarding successive approximations to it. Vijaykumar Gullapalli and Andrew Barto (1992) and Marco Dorigo and Marco Colombetti (1994) applied shaping to machine learning; Jette Randløv and Preben Alstrøm's bicycle experiment (1998) exposed its pitfalls; and Andrew Ng, Daishi Harada and Stuart Russell proved the potential-based shaping theorem (ICML 1999).
Usage note
In animal training and early RL, shaping often meant gradually increasing task difficulty — closer to today's curriculum learningcurriculum learning, n.: Teaching a model the way schools teach people: starting with easy examples and gradually moving to harder ones, instead of presenting everything in random order. — rather than altering the reward itself.
Related
- reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do.
- reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
- intrinsic motivationintrinsic motivation, n.: Giving an AI agent an inner drive — such as curiosity about surprising things — so it keeps exploring and learning even when the task itself offers no reward yet.
- curriculum learningcurriculum learning, n.: Teaching a model the way schools teach people: starting with easy examples and gradually moving to harder ones, instead of presenting everything in random order.
- specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.
- process reward modelprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof.
- sparse rewardsparse reward, n.: A situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success.
Sources
- Ng, A. Y., Harada, D. & Russell, S. (1999). Policy invariance under reward transformations: Theory and application to reward shaping. Proceedings of ICML 1999.
- Randløv, J. & Alstrøm, P. (1998). Learning to Drive a Bicycle using Reinforcement Learning and Shaping. Proceedings of ICML 1998.
Mentioned in
rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time., reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., curriculum learningcurriculum learning, n.: Teaching a model the way schools teach people: starting with easy examples and gradually moving to harder ones, instead of presenting everything in random order., reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do., sparse rewardsparse reward, n.: A situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success., intrinsic motivationintrinsic motivation, n.: Giving an AI agent an inner drive — such as curiosity about surprising things — so it keeps exploring and learning even when the task itself offers no reward yet.