Skip to content
Last Dictionary
Plane
Reinforcement Learning

trust region policy optimization

nounAlso: TRPO

In plain wordsA method for improving an AI's strategy in careful steps, never changing it too much at once, so that a single bad update cannot wreck what it has already learned.

  1. A policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely. algorithm that maximizes a surrogate objective — the expected advantageadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse. of the new policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards., estimated from data collected with the old one through importance-samplingimportance sampling, n.: A shortcut for estimating averages: draw samples from a convenient source, then reweight each one to correct for how much more or less likely it would be under the source you actually care about. ratios — subject to a limit on the average Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right. between old and new policies. The limit defines a 'trust region' within which the surrogate is a reliable guide, guarding against the destructive updates that plague plain policy gradients. Its derivation starts from a bound under which every update is guaranteed to improve the policy and then relaxes it into a workable procedure: the update direction is a natural gradient, computed with conjugate-gradient solves using Fisher-vector products, and the step size is set by a line search. The original paper applied it to simulated locomotion and to Atari games played from screen images, with little hyperparameter tuning, but its second-order machinery is complex and awkward with networks that share parameters, which motivated the simpler proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback..

    TRPO's KL limit of 0.01 kept the walker from collapsing after a lucky batch.

max⁡θ  E[πθ(a∣s)πθold(a∣s) A^(s,a)]s.t.E[DKL(πθold(⋅∣s) ∥ πθ(⋅∣s))]≤δ\max_\theta\; \mathbb{E}\left[ \frac{\pi_\theta(a \mid s)}{\pi_{\theta_{\text{old}}}(a \mid s)}\, \hat{A}(s,a) \right] \quad \text{s.t.} \quad \mathbb{E}\left[ D_{\mathrm{KL}}\left( \pi_{\theta_{\text{old}}}(\cdot \mid s) \,\|\, \pi_\theta(\cdot \mid s) \right) \right] \le \delta

Origin

Introduced by John Schulman, Sergey Levine, Philipp Moritz, Michael Jordan and Pieter Abbeel at UC Berkeley in 'Trust Region Policy Optimization' (arXiv, February 2015; ICML 2015), building on Sham Kakade's natural policy gradient (2001) and Kakade and John Langford's conservative policy iteration (2002). 'Trust region' is borrowed from numerical optimization.

Related

See also
  • policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely.
  • Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right.
  • advantage functionadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse.
  • generalized advantage estimationgeneralized advantage estimation, n.: A recipe for judging how much better each action was than expected by blending short-term and long-term evidence, trading a little accuracy for much steadier learning.
  • actor–criticactor–critic, n.: A way of training an AI agent in which one part, the 'actor', makes the moves while another, the 'critic', learns to predict how well they will work out, and the actor improves by heeding the critic.
Contrast with
  • proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.

Sources

  1. Schulman, J., Levine, S., Moritz, P., Jordan, M. I. & Abbeel, P. (2015). Trust Region Policy Optimization. ICML 2015. arXiv:1502.05477. arxiv.org/abs/1502.05477
  2. Kakade, S. (2001). A Natural Policy Gradient. Advances in Neural Information Processing Systems 14.

Mentioned in

deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images., John SchulmanJohn Schulman, prop. n.: An American AI researcher who invented widely used reinforcement-learning algorithms such as PPO, co-founded OpenAI, and led the post-training work behind ChatGPT., Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right., on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes., policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely., proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback., generalized advantage estimationgeneralized advantage estimation, n.: A recipe for judging how much better each action was than expected by blending short-term and long-term evidence, trading a little accuracy for much steadier learning., KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer., Pieter AbbeelPieter Abbeel, prop. n.: A Belgian-American roboticist at Berkeley whose research taught robots to learn from demonstrations and from trial and error, and who co-founded the robotics company Covariant., Sergey LevineSergey Levine, prop. n.: A Berkeley computer scientist, one of the most prolific researchers on how robots can learn skills from data and experience, and a co-founder of the robotics company Physical Intelligence.