Skip to content
Last Dictionary
Plane
Reinforcement LearningTraining & Post-trainingLanguage Models & Transformers

proximal policy optimization

nounAlso: PPO, PPO-Clip, PPO-Penalty, clipped surrogate objective

In plain wordsA widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.

  1. A policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely. algorithm that keeps each update close to the previous policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards. with a simple first-order objective instead of the hard constraint of TRPOtrust region policy optimization, n.: A method for improving an AI's strategy in careful steps, never changing it too much at once, so that a single bad update cannot wreck what it has already learned.. Its main form, PPO-Clip, multiplies the advantageadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse. estimate by the probability ratio between new and old policies, clips that ratio to [1 − ε, 1 + ε] (typically ε = 0.1–0.2) and takes the more pessimistic of the clipped and unclipped terms, removing the incentive for large policy changes. This lets PPO run several epochs of minibatch gradient ascent on each batch of on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes. rolloutsrollout, n.: Letting an AI play out a task with its current strategy, from some starting point to the end, to see how things turn out and gather data for learning or planning., usually with generalized advantage estimationgeneralized advantage estimation, n.: A recipe for judging how much better each action was than expected by blending short-term and long-term evidence, trading a little accuracy for much steadier learning., a value-function loss and an entropy bonusentropy regularization, n.: Rewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early.. Simple and robust, it became the default deep RL algorithm (OpenAI Five used it for Dota 2) and the standard optimizer of reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., with a KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer. toward a reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started., for InstructGPT and ChatGPT. Since 2024, critic-free variants such as GRPOgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average. have become popular alternatives for training reasoning modelsreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning..

    PPO training diverged until we normalized advantages and lowered the clip range to 0.1.

LCLIP(θ)=Et[min⁡(ρt(θ)A^t,  clip⁡(ρt(θ),1−ϵ,1+ϵ)A^t)],ρt(θ)=πθ(at∣st)πθold(at∣st)L^{\text{CLIP}}(\theta) = \mathbb{E}_t\left[ \min\left( \rho_t(\theta)\hat{A}_t,\; \operatorname{clip}\left(\rho_t(\theta), 1-\epsilon, 1+\epsilon\right)\hat{A}_t \right) \right], \quad \rho_t(\theta) = \frac{\pi_\theta(a_t \mid s_t)}{\pi_{\theta_{\text{old}}}(a_t \mid s_t)}

Origin

Introduced by John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford and Oleg Klimov of OpenAI in 'Proximal Policy Optimization Algorithms' (arXiv, 20 July 2017), with a blog post and an implementation in OpenAI Baselines. OpenAI used it to fine-tune language models from human preferences from 2019, and its use for InstructGPT (2022) and ChatGPT carried it from games and robotics into mainstream language-model training.

Usage note

Results are notoriously sensitive to implementation details — advantage normalization, value clipping, learning-rate annealing, observation normalization — as documented by Logan Engstrom and colleagues (2020) and in Shengyi Huang and colleagues' catalog of 37 PPO implementation details (2022).

Related

See also
  • trust region policy optimizationtrust region policy optimization, n.: A method for improving an AI's strategy in careful steps, never changing it too much at once, so that a single bad update cannot wreck what it has already learned.
  • policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely.
  • generalized advantage estimationgeneralized advantage estimation, n.: A recipe for judging how much better each action was than expected by blending short-term and long-term evidence, trading a little accuracy for much steadier learning.
  • reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
  • group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average.
  • KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer.
  • actor–criticactor–critic, n.: A way of training an AI agent in which one part, the 'actor', makes the moves while another, the 'critic', learns to predict how well they will work out, and the actor improves by heeding the critic.
Contrast with
  • direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely.

Sources

  1. Schulman, J., Wolski, F., Dhariwal, P., Radford, A. & Klimov, O. (2017). Proximal Policy Optimization Algorithms. arXiv:1707.06347. arxiv.org/abs/1707.06347
  2. Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022.
  3. Engstrom, L. et al. (2020). Implementation Matters in Deep RL: A Case Study on PPO and TRPO. ICLR 2020.

Mentioned in

reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., actor–criticactor–critic, n.: A way of training an AI agent in which one part, the 'actor', makes the moves while another, the 'critic', learns to predict how well they will work out, and the actor improves by heeding the critic., advantage functionadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse., Alec RadfordAlec Radford, prop. n.: An American AI researcher who led OpenAI's first GPT language models and its CLIP and Whisper systems, key steps toward today's chatbots and multimodal AI., deep reinforcement learningdeep reinforcement learning, n.: Reinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images., direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely., group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average., InstructGPTInstructGPT, prop. n.: A 2022 OpenAI language model trained with human feedback to follow instructions; people preferred its answers to those of a model 100 times larger, and its recipe led directly to ChatGPT., John SchulmanJohn Schulman, prop. n.: An American AI researcher who invented widely used reinforcement-learning algorithms such as PPO, co-founded OpenAI, and led the post-training work behind ChatGPT., model-free reinforcement learningmodel-free reinforcement learning, n.: Learning to act well purely from trial and error, without ever building an internal picture of how the world works — like learning a game by habit rather than by thinking moves ahead., on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes., policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely., reinforcement fine-tuningreinforcement fine-tuning, n.: Improving a trained AI model by trial and error instead of examples to copy: it tries answers, a grader scores them, and the model shifts toward whatever earned high scores., reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds., rejection samplingrejection sampling, n.: Generating many candidate answers and keeping only those that pass a check — a scoring model, a test or a correct final answer — often so the AI can then be trained on its own best work., RL scalingRL scaling, n.: The practice, and the finding behind it, of giving AI models far more trial-and-error practice after their initial training, especially on problems with checkable answers, which has so far improved their reasoning fairly predictably., rolloutrollout, n.: Letting an AI play out a task with its current strategy, from some starting point to the end, to see how things turn out and gather data for learning or planning., A3CA3C, abbr.: A 2016 DeepMind training method in which many copies of an AI agent learn in parallel on separate copies of a task, pooling their updates into one shared neural network., direct alignment algorithmdirect alignment algorithm, n.: Any method that tunes an AI directly on examples of preferred and dispreferred answers, skipping the separate scoring model and trial-and-error training used in classic RLHF., entropy regularizationentropy regularization, n.: Rewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early., generalized advantage estimationgeneralized advantage estimation, n.: A recipe for judging how much better each action was than expected by blending short-term and long-term evidence, trading a little accuracy for much steadier learning., importance samplingimportance sampling, n.: A shortcut for estimating averages: draw samples from a convenient source, then reweight each one to correct for how much more or less likely it would be under the source you actually care about., KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer., OpenAI FiveOpenAI Five, prop. n.: A team of five AI players built by OpenAI that learned the team video game Dota 2 by playing against itself, and in April 2019 beat the reigning world-champion human team.