discount factor
In plain wordsA number between 0 and 1 that makes an AI agent value future rewards less than immediate ones — like preferring a dollar today to a dollar next year.
The parameter γ ∈ [0, 1] by which a rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time. received k steps in the future is multiplied (as γ^k) when computing the returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible.. It plays three roles: it keeps returns finite in tasks that never terminate; it encodes a preference for sooner rewards, with 1/(1 − γ) acting as a rough effective horizon (γ = 0.99 looks about 100 steps ahead); and it reduces the variance of value estimates at the cost of bias, which is why it is often tuned as a hyperparameter rather than treated as part of the task. With γ = 0 the agent is myopic and cares only about the next reward; as γ approaches 1, distant consequences count almost fully and learning becomes slower and noisier. Convergence guarantees for the Bellman equationBellman equation, n.: A rule saying that how good a situation is equals the reward you get now plus how good the next situation is, slightly discounted — so long-term planning can be solved one step at a time. in infinite-horizon problems rely on γ < 1.
Raising the discount factor from 0.9 to 0.995 let the agent learn to save fuel for the final climb.
Origin
Exponential discounting comes from economics — Paul Samuelson's discounted-utility model (1937) — and was built into the infinite-horizon sequential decision models of Richard Bellman (1950s), Ronald Howard (1960) and David Blackwell, whose 1965 paper 'Discounted Dynamic Programming' gave the theory its standard form.
Usage note
In large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. RL, γ is often set to 1 because each response is a short episode with a single final reward. People and animals tend to discount hyperbolically rather than exponentially, a difference studied in behavioral economics.
Related
- returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible.
- rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time.
- value functionvalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board.
- Bellman equationBellman equation, n.: A rule saying that how good a situation is equals the reward you get now plus how good the next situation is, slightly discounted — so long-term planning can be solved one step at a time.
- Markov decision processMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice.
- generalized advantage estimationgeneralized advantage estimation, n.: A recipe for judging how much better each action was than expected by blending short-term and long-term evidence, trading a little accuracy for much steadier learning.
Sources
- Blackwell, D. (1965). Discounted Dynamic Programming. Annals of Mathematical Statistics 36(1), 226–235.
Mentioned in
Bellman equationBellman equation, n.: A rule saying that how good a situation is equals the reward you get now plus how good the next situation is, slightly discounted — so long-term planning can be solved one step at a time., Markov decision processMarkov decision process, n.: A mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice., returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible., reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do., generalized advantage estimationgeneralized advantage estimation, n.: A recipe for judging how much better each action was than expected by blending short-term and long-term evidence, trading a little accuracy for much steadier learning., myopiamyopia, n.: Designing an AI to care only about the immediate results of its actions, not about what happens far in the future, so it has no reason for long-term scheming.