Skip to content
Last Dictionary
Plane
Training & Post-trainingReinforcement LearningLanguage Models & Transformers

REINFORCE leave-one-out

nounAlso: RLOO, leave-one-out baseline

In plain wordsA simple reinforcement-learning method that has a model answer each question several times and judges each answer against the average score of the other answers.

  1. A variant of the REINFORCEREINFORCE, n.: The simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly. policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely. estimator that draws k samples per input and uses, as the baseline for each sample, the mean reward of the other k − 1 samples. The leave-one-out baseline keeps the gradient estimate unbiased, because it does not depend on the sample it corrects, and reduces variance without a learned value function (critic). In language-model post-training each whole response is treated as a single action, rewarded by a reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. or verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded., which suits RLHF and RLVR, where reward arrives only at the end of a sequence. RLOO is cheaper and simpler than proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback. and was reported to match or beat it, and offline methods such as direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely., in RLHF experiments. It is closely related to group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average., which normalizes by the group mean (including the sample itself) and standard deviation.

    We replaced PPO with RLOO at k = 4 and dropped the value model entirely.

∇θJ≈1k∑i=1k(r(yi)−1k−1∑j≠ir(yj))∇θlog⁡πθ(yi∣x)\nabla_\theta J \approx \frac{1}{k} \sum_{i=1}^{k} \Big( r(y_i) - \frac{1}{k-1} \sum_{j \ne i} r(y_j) \Big) \nabla_\theta \log \pi_\theta(y_i \mid x)

Origin

The leave-one-out baseline for multi-sample REINFORCE was proposed by Wouter Kool, Herke van Hoof and Max Welling in 2019. Arash Ahmadian and colleagues at Cohere revived it for language models in February 2024, arguing that many components of PPO are unnecessary in RLHF. REINFORCE itself is due to Ronald Williams (1987, fully described 1992).

Related

See also
  • REINFORCEREINFORCE, n.: The simplest way to improve an AI's strategy by trial and error: after each attempt, make every choice in it more likely if the attempt went well and less likely if it went badly.
  • policy gradientpolicy gradient, n.: A family of methods that improve an AI's strategy directly: make actions that led to good results a bit more likely, and actions that led to bad results a bit less likely.
  • group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average.
  • proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.
  • reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
  • advantage functionadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse.
Contrast with
  • proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.

Sources

  1. Ahmadian, A. et al. (2024). Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. arXiv:2402.14740. arxiv.org/abs/2402.14740
  2. Kool, W., van Hoof, H. & Welling, M. (2019). Buy 4 REINFORCE Samples, Get a Baseline for Free! ICLR 2019 workshop on Deep RL Meets Structured Prediction.

Mentioned in

group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average.