sparse reward
In plain wordsA situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success.
A reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do. that is zero, or constant, for almost all transitions and informative only at rare events such as reaching a goal, winning a game or passing a test. Sparse rewards are often the most faithful way to specify a task — they state what is wanted without dictating how — but they make learning hard: a randomly exploring agent may never observe a nonzero reward, leaving nothing to learn from, and when success does occur the credit assignment problemcredit assignment problem, n.: The puzzle of figuring out which of many earlier decisions, or which parts of a system, deserve the credit or blame for a final outcome — like working out which moves won a chess game. of which earlier actions caused it is severe. Remedies include reward shapingreward shaping, n.: Adding extra hints to an AI agent's scoring — small rewards for steps in the right direction — so it learns faster than it would from the final outcome alone., curiosityintrinsic motivation, n.: Giving an AI agent an inner drive — such as curiosity about surprising things — so it keeps exploring and learning even when the task itself offers no reward yet. and count-based exploration, demonstrations and imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error., curriculacurriculum learning, n.: Teaching a model the way schools teach people: starting with easy examples and gradually moving to harder ones, instead of presenting everything in random order., goal relabeling (hindsight experience replay) and hierarchical reinforcement learninghierarchical reinforcement learning, n.: Reinforcement learning organized in layers, where a high-level part chooses goals or skills — 'go to the kitchen' — and lower-level parts handle the detailed actions to achieve them.. In large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. training, pass/fail rewards from unit tests or answer checkers are sparse but workable, because a pretrained model already succeeds on some fraction of attempts.
The reward is sparse — +1 only when the block reaches the shelf — so we seeded training with demonstrations.
Origin
A descriptive term rather than a coinage, it became common in deep RL in the mid-2010s, when Atari games such as Montezuma's Revenge, in which a randomly acting agent almost never scores, became the canonical hard-exploration benchmarks. The underlying difficulty of delayed reward had been central to RL since Marvin Minsky's 1961 discussion of credit assignment.
Related
- rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time.
- credit assignment problemcredit assignment problem, n.: The puzzle of figuring out which of many earlier decisions, or which parts of a system, deserve the credit or blame for a final outcome — like working out which moves won a chess game.
- intrinsic motivationintrinsic motivation, n.: Giving an AI agent an inner drive — such as curiosity about surprising things — so it keeps exploring and learning even when the task itself offers no reward yet.
- exploration–exploitation trade-offexploration–exploitation trade-off, n.: The choice between trying something new that might turn out better and sticking with what already works — like picking a familiar restaurant versus testing a new one.
- reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.
- hierarchical reinforcement learninghierarchical reinforcement learning, n.: Reinforcement learning organized in layers, where a high-level part chooses goals or skills — 'go to the kitchen' — and lower-level parts handle the detailed actions to achieve them.
- reward shapingreward shaping, n.: Adding extra hints to an AI agent's scoring — small rewards for steps in the right direction — so it learns faster than it would from the final outcome alone.
Mentioned in
rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time., credit assignment problemcredit assignment problem, n.: The puzzle of figuring out which of many earlier decisions, or which parts of a system, deserve the credit or blame for a final outcome — like working out which moves won a chess game., curriculum learningcurriculum learning, n.: Teaching a model the way schools teach people: starting with easy examples and gradually moving to harder ones, instead of presenting everything in random order., episodeepisode, n.: One complete run of a task from start to finish — a single game of chess, one attempt at a maze — after which everything resets and the agent tries again., epsilon-greedyepsilon-greedy, adj.: A simple rule for balancing trying new things against using what works: most of the time take the best-known option, but a small fraction of the time pick at random., exploration–exploitation trade-offexploration–exploitation trade-off, n.: The choice between trying something new that might turn out better and sticking with what already works — like picking a familiar restaurant versus testing a new one., goal-conditioned reinforcement learninggoal-conditioned reinforcement learning, n.: Training one AI agent to reach many different goals — whichever target it is given — instead of a single fixed task, so a robot can, say, place an object anywhere asked., hierarchical reinforcement learninghierarchical reinforcement learning, n.: Reinforcement learning organized in layers, where a high-level part chooses goals or skills — 'go to the kitchen' — and lower-level parts handle the detailed actions to achieve them., intrinsic motivationintrinsic motivation, n.: Giving an AI agent an inner drive — such as curiosity about surprising things — so it keeps exploring and learning even when the task itself offers no reward yet., reward shapingreward shaping, n.: Adding extra hints to an AI agent's scoring — small rewards for steps in the right direction — so it learns faster than it would from the final outcome alone.