Skip to content
Last Dictionary
Plane
Reinforcement LearningTraining & Post-training

policy gradient

nounAlso: policy gradient method, policy gradient theorem, score-function estimator, likelihood-ratio gradient

In plain wordsA family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely.

  1. A class of reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. methods that optimize a parametrized policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards. π_θ directly by gradient ascent on its expected returnreturn, n.: The total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible. J(θ). The policy gradient theorem shows that this gradient can be estimated from sampled trajectoriestrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received. without differentiating through the environment: it is the expectation of the gradient of each action's log-probability, weighted by how good the action turned out to be — the return in REINFORCEREINFORCE, n.: The simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly., or a lower-variance advantageadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse. estimate in actor–criticactor–critic, n.: A way of training an AI agent in which one part, the 'actor', makes the moves while another, the 'critic', learns to predict how well they will work out, and the actor improves by heeding the critic. methods. Policy gradients handle continuous and very large action spacesaction space, n.: The full menu of moves an AI agent is allowed to make — a handful of buttons in a video game, or any combination of motor forces for a robot arm. and stochastic policies naturally and change the policy smoothly, but their estimates are noisy and the methods are typically on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes. and sample-hungry. Variance reduction (baselines, generalized advantage estimationgeneralized advantage estimation, n.: A recipe for judging how much better each action was than expected by blending short-term and long-term evidence, trading a little accuracy for much steadier learning.) and step-size control (TRPOtrust region policy optimization, n.: A method for improving an AI's strategy in careful steps, never changing it too much at once, so that a single bad update cannot wreck what it has already learned., PPOproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.) make them practical. Most RL fine-tuning of language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., including GRPOgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average., is policy-gradient optimization over token sequences.

    The policy gradient pushes up the log-probability of every token in responses that scored above the baseline.

∇θJ(θ)=Eπθ[∑t∇θlog⁡πθ(At∣St) A^t]\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta}\left[ \sum_{t} \nabla_\theta \log \pi_\theta(A_t \mid S_t)\, \hat{A}_t \right]

Origin

Ronald Williams's REINFORCE algorithms (1987 conference paper; Machine Learning, 1992) gave the first general likelihood-ratio policy gradient for neural networks; the same estimator was developed for simulation optimization by Peter Glynn and others in the 1980s. Richard Sutton, David McAllester, Satinder Singh and Yishay Mansour proved the policy gradient theorem with function approximation (NIPS 1999), as did Vijay Konda and John Tsitsiklis for actor–critic methods.

Related

See also
  • REINFORCEREINFORCE, n.: The simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly.
  • actor–criticactor–critic, n.: A way of training an AI agent in which one part, the 'actor', makes the moves while another, the 'critic', learns to predict how well they will work out, and the actor improves by heeding the critic.
  • proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.
  • trust region policy optimizationtrust region policy optimization, n.: A method for improving an AI's strategy in careful steps, never changing it too much at once, so that a single bad update cannot wreck what it has already learned.
  • advantage functionadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse.
  • generalized advantage estimationgeneralized advantage estimation, n.: A recipe for judging how much better each action was than expected by blending short-term and long-term evidence, trading a little accuracy for much steadier learning.
  • group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average.
Contrast with
  • Q-learningQ-learning, n.: A trial-and-error method in which an agent keeps a score for every move in every situation and nudges each score toward the reward it got plus the best score available next.

Sources

  1. Williams, R. J. (1992). Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine Learning 8, 229–256. link.springer.com/article/10.1007/BF00992696
  2. Sutton, R. S., McAllester, D., Singh, S. & Mansour, Y. (2000). Policy Gradient Methods for Reinforcement Learning with Function Approximation. Advances in Neural Information Processing Systems 12 (NIPS 1999).

Mentioned in

reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., action spaceaction space, n.: The full menu of moves an AI agent is allowed to make — a handful of buttons in a video game, or any combination of motor forces for a robot arm., actor–criticactor–critic, n.: A way of training an AI agent in which one part, the 'actor', makes the moves while another, the 'critic', learns to predict how well they will work out, and the actor improves by heeding the critic., advantage functionadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse., baselinebaseline, n.: A simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value., group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average., John SchulmanJohn Schulman, prop. n.: An American AI researcher who invented widely used reinforcement-learning algorithms such as PPO, co-founded OpenAI, and led the post-training work behind ChatGPT., model-free reinforcement learningmodel-free reinforcement learning, n.: Learning to act well purely from trial and error, without ever building an internal picture of how the world works — like learning a game by habit rather than by thinking moves ahead., on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes., policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards., proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback., reinforcement fine-tuningreinforcement fine-tuning, n.: Improving a trained AI model by trial and error instead of examples to copy: it tries answers, a grader scores them, and the model shifts toward whatever earned high scores., Richard SuttonRichard Sutton, prop. n.: A Canadian computer scientist who, with Andrew Barto, laid the groundwork for teaching machines by trial, error and reward, and whose essay 'The Bitter Lesson' argues that methods that scale with computing power always win., trajectorytrajectory, n.: The record of what happened as an AI agent acted: the sequence of situations it was in, the actions it took and the rewards it received., value functionvalue function, n.: An estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board., decision transformerdecision transformer, n.: A way of solving decision problems with the same kind of model that powers chatbots: it reads past situations, actions and the score it is asked to achieve, and predicts the next action., entropy regularizationentropy regularization, n.: Rewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early., generalized advantage estimationgeneralized advantage estimation, n.: A recipe for judging how much better each action was than expected by blending short-term and long-term evidence, trading a little accuracy for much steadier learning., ReflexionReflexion, prop. n.: A technique in which an AI agent that fails at a task writes itself a short note on what went wrong, then reads that note before trying again — learning from mistakes without retraining., REINFORCEREINFORCE, n.: The simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly., REINFORCE leave-one-outREINFORCE leave-one-out, n.: A simple reinforcement-learning method that has a model answer each question several times and judges each answer against the average score of the other answers., reparameterization trickreparameterization trick, n.: A trick that rewrites a random draw as fixed numbers plus separate, outside randomness, so that a neural network can still learn through a step that involves chance., self-taught reasonerself-taught reasoner, n.: A training method in which a language model writes step-by-step explanations for problems, keeps the ones that led to correct answers, and learns from them, so it gradually teaches itself to reason., trust region policy optimizationtrust region policy optimization, n.: A method for improving an AI's strategy in careful steps, never changing it too much at once, so that a single bad update cannot wreck what it has already learned.