Skip to content
Last Dictionary
Plane
Training & Post-trainingSafety & AlignmentReinforcement Learning

reward overoptimization

nounAlso: reward model overoptimization, reward over-optimization, overoptimization, reward overoptimisation

In plain wordsWhat happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse.

  1. The growing gap between a proxy reward and the true objective as a policy is optimized more strongly against the proxy, typically a learned reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.. Early in training, proxy and true quality rise together; past some point the proxy keeps rising while true quality — judged by people or a held-out 'gold' reward model — plateaus and then declines, as the policy exploits the proxy's errors, for instance with verbose, flattering or formulaic answers. Leo Gao and colleagues found that gold reward follows predictable curves in the square root of the KL divergence from the initial policy, with different functional forms for best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers. and RL, and that larger reward models overoptimize more slowly. Mitigations include the KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer., early stopping, reward-model ensembles, refreshing reward models with new on-policy labels, and replacing learned rewards with checkable ones. The effect also afflicts direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely. and other direct alignment algorithmsdirect alignment algorithm, n.: Any method that tunes an AI directly on examples of preferred and dispreferred answers, skipping the separate scoring model and trial-and-error training used in classic RLHF..

    Gold-model scores peaked around 8 nats of KL; everything after that was reward overoptimization.

RRL(d)=d (αRL−βRLlog⁡d),d=KL(π ∥ πinit)R_{\mathrm{RL}}(d) = d\,(\alpha_{\mathrm{RL}} - \beta_{\mathrm{RL}} \log d), \qquad d = \sqrt{\mathrm{KL}(\pi \,\|\, \pi_{\mathrm{init}})}

Origin

The effect was observed in OpenAI's RLHF work on summarization (Nisan Stiennon and colleagues, 2020), where optimizing too far against the reward model degraded summaries. Leo Gao, John Schulman and Jacob Hilton quantified it in October 2022 with a synthetic gold-reward setup, in a paper whose title popularized the term; Rafael Rafailov and colleagues (2024) showed similar degradation in direct alignment algorithms.

Usage note

A special case of Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure. and closely related to reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., which usually denotes exploiting a specific flaw in a reward, whereas overoptimization describes the aggregate drift away from true quality.

Related

See also
  • reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
  • Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure.
  • reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.
  • KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer.
  • specification gamingspecification gaming, n.: When an AI meets the letter of its goal while defeating its purpose, like a genie who grants exactly the words of a wish instead of what the wisher meant.
  • best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers.
  • sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful.
Contrast with
  • reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.

Sources

  1. Gao, L., Schulman, J. & Hilton, J. (2022). Scaling Laws for Reward Model Overoptimization. ICML 2023. arXiv:2210.10760. arxiv.org/abs/2210.10760
  2. Rafailov, R. et al. (2024). Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms. arXiv:2406.02900. arxiv.org/abs/2406.02900

Mentioned in

reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers., Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure., reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples., verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded., direct alignment algorithmdirect alignment algorithm, n.: Any method that tunes an AI directly on examples of preferred and dispreferred answers, skipping the separate scoring model and trial-and-error training used in classic RLHF., KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer., mild optimizationmild optimization, n.: Building an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted., RewardBenchRewardBench, prop. n.: A public test that checks whether the scoring models used to train chatbots can tell a better answer from a worse one, by counting how often they prefer the right response.