group relative policy optimization
In plain wordsA reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average.
A policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely. algorithm for language models, a variant of proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback. that drops the learned value network (critic). For each prompt it samples a group of G responses, scores each with a reward (often a rule-based check, as in reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.), and gives every token of a response the same advantageadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse.: the response's reward minus the group mean, divided by the group standard deviation. The policy is updated with PPO's clipped importance-ratio objective, and a KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer. toward a reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started. is added to the loss rather than to the reward. Removing the critic saves substantial memory and avoids training a value model on long sequences, at the cost of several samples per prompt; prompts on which all samples succeed or all fail yield no signal. GRPO became the default for open reasoning-model training after DeepSeek-R1DeepSeek-R1, prop. n.: A free, downloadable AI reasoning model released by the Chinese company DeepSeek in January 2025. It matched leading American models at math and coding for a fraction of the reported cost, shaking the industry.. Successors include Dr. GRPO, which removes length and difficulty biases, DAPO, and GSPO with sequence-level importance ratios.
With GRPO we sample 16 solutions per problem, so a batch where all 16 fail teaches the model nothing.
Origin
Introduced by Zhihong Shao and colleagues at DeepSeek in the DeepSeekMath paper (February 2024), whose 7-billion-parameter model reached 51.7 percent on the MATH benchmark, as a memory-saving variant of PPO. It drew wide attention when DeepSeek used it to train DeepSeek-R1-Zero and DeepSeek-R1 (January 2025), and open-source RL libraries adopted it within weeks.
Usage note
Group-relative baselines resemble earlier multi-sample REINFORCEREINFORCE, n.: The simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly. baselines such as REINFORCE leave-one-outREINFORCE leave-one-out, n.: A simple reinforcement-learning method that has a model answer each question several times and judges each answer against the average score of the other answers.. Critics showed that GRPO's normalization by response length and by reward standard deviation introduces biases that can inflate the length of wrong answers (Zichen Liu and colleagues, 2025).
Related
- proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.
- reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.
- policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely.
- advantage functionadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse.
- DeepSeek-R1DeepSeek-R1, prop. n.: A free, downloadable AI reasoning model released by the Chinese company DeepSeek in January 2025. It matched leading American models at math and coding for a fraction of the reported cost, shaking the industry.
- REINFORCE leave-one-outREINFORCE leave-one-out, n.: A simple reinforcement-learning method that has a model answer each question several times and judges each answer against the average score of the other answers.
- KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer.
- proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.
- direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely.
Sources
- Shao, Z. et al. (2024). DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300. arxiv.org/abs/2402.03300
- Liu, Z. et al. (2025). Understanding R1-Zero-Like Training: A Critical Perspective. arXiv:2503.20783. arxiv.org/abs/2503.20783
- Zheng, C. et al. (2025). Group Sequence Policy Optimization. arXiv:2507.18071. arxiv.org/abs/2507.18071
Mentioned in
DeepSeekDeepSeek, prop. n.: A Chinese artificial intelligence company whose free, cheaply trained models startled the world in January 2025 by nearly matching the best American systems., actor–criticactor–critic, n.: A way of training an AI agent in which one part, the 'actor', makes the moves while another, the 'critic', learns to predict how well they will work out, and the actor improves by heeding the critic., advantage functionadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse., baselinebaseline, n.: A simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value., DeepSeek-R1DeepSeek-R1, prop. n.: A free, downloadable AI reasoning model released by the Chinese company DeepSeek in January 2025. It matched leading American models at math and coding for a fraction of the reported cost, shaking the industry., policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely., proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback., reinforcement fine-tuningreinforcement fine-tuning, n.: Improving a trained AI model by trial and error instead of examples to copy: it tries answers, a grader scores them, and the model shifts toward whatever earned high scores., reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds., RL scalingRL scaling, n.: The practice, and the finding behind it, of giving AI models far more trial-and-error practice after their initial training, especially on problems with checkable answers, which has so far improved their reasoning fairly predictably., entropy collapseentropy collapse, n.: A failure in reinforcement-learning training where an AI quickly becomes overconfident and stops trying different answers, so it explores less and its improvement stalls., importance samplingimportance sampling, n.: A shortcut for estimating averages: draw samples from a convenient source, then reweight each one to correct for how much more or less likely it would be under the source you actually care about., reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started., REINFORCEREINFORCE, n.: The simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly., REINFORCE leave-one-outREINFORCE leave-one-out, n.: A simple reinforcement-learning method that has a model answer each question several times and judges each answer against the average score of the other answers.