Skip to content
Last Dictionary
Plane
Training & Post-trainingReinforcement LearningLanguage Models & Transformers

reward model

nounAlso: RM, preference model, PM, reward modeling, learned reward model

In plain wordsA separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.

  1. 1

    Post-training

    A learned function that maps an input and a candidate output to a scalar score approximating human (or AI) preference, used as the rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time. signal when reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. cannot query people directly. For language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. it is usually a copy of a pretrained or fine-tuned model whose unembeddingunembedding, n.: The last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores. is replaced by a scalar head, trained on preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like. with a Bradley–TerryBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths. objective: the loss is the negative log-sigmoid of the score gap between preferred and rejected responses. Reward models drive RLHFreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., rank samples in best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers. and rejection samplingrejection sampling, n.: Generating many candidate answers and keeping only those that pass a check — a scoring model, a test or a correct final answer — often so the AI can then be trained on its own best work., and filter training data. Variants include outcomeoutcome reward model, n.: A model that judges only whether an AI's final answer is right, ignoring how it got there — like a teacher who checks just the answer box. and processprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof. reward models for reasoning, and generative reward models that write a critique or verdict before scoring. Because any reward model is an imperfect proxy, optimizing hard against it produces reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse..

    The reward model preferred longer answers, so the policy learned to pad its responses.

  2. 2

    Model-based reinforcement learning

    The component of a learned environment model (a world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real.) that predicts the immediate reward for a given state and action, used with a learned transition model for planning or for training on imagined rolloutsrollout, n.: Letting an AI play out a task with its current strategy, from some starting point to the end, to see how things turn out and gather data for learning or planning..

LRM(ϕ)=− E(x, yw, yl)[log⁡σ(rϕ(x,yw)−rϕ(x,yl))]\mathcal{L}_{\mathrm{RM}}(\phi) = -\,\mathbb{E}_{(x,\,y_w,\,y_l)}\left[\log \sigma\big(r_\phi(x, y_w) - r_\phi(x, y_l)\big)\right]

Origin

Learning a reward function from people has roots in inverse reinforcement learning and in the 2017 preference-learning work of Paul Christiano and colleagues. Jan Leike and colleagues at DeepMind (2018) proposed 'reward modeling' as a research program for agent alignment, and Daniel Ziegler and colleagues at OpenAI (2019) trained reward models on human comparisons of language-model text. Anthropic's papers call the same object a preference model.

Usage note

A reward model scores outputs; it is not the hand-written reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do. of a classical RL environment, though it plays that role. Documented biases include favoring longer, more confident or more flattering answers, which can feed sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful..

Related

See also
  • reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
  • preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like.
  • Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths.
  • process reward modelprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof.
  • outcome reward modeloutcome reward model, n.: A model that judges only whether an AI's final answer is right, ignoring how it got there — like a teacher who checks just the answer box.
  • verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded.
  • reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse.
  • LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.
Contrast with
  • reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do.
  • verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded.

Sources

  1. Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V. & Legg, S. (2018). Scalable agent alignment via reward modeling: a research direction. arXiv:1811.07871. arxiv.org/abs/1811.07871
  2. Ziegler, D. M. et al. (2019). Fine-Tuning Language Models from Human Preferences. arXiv:1909.08593. arxiv.org/abs/1909.08593

Mentioned in

reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time., reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers., Constitutional AIConstitutional AI, n.: Anthropic's technique for making an AI less harmful by having it check and rewrite its own answers against a short written set of principles, so that people need not label harmful examples by hand., direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely., Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure., InstructGPTInstructGPT, prop. n.: A 2022 OpenAI language model trained with human feedback to follow instructions; people preferred its answers to those of a model 100 times larger, and its recipe led directly to ChatGPT., instruction datasetinstruction dataset, n.: A collection of example requests paired with good answers, used to teach a pretrained language model to follow instructions and act as a helpful assistant., LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers., outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want., Paul ChristianoPaul Christiano, prop. n.: An American AI-alignment researcher who led the work that made training AI from human preferences practical, the basis of how chatbots are taught to be helpful., policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards., preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like., reinforcement learning from AI feedbackreinforcement learning from AI feedback, n.: Training an AI with preference judgments made by another AI instead of by people, which is cheaper and faster but only as good as the AI doing the judging., reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds., rejection samplingrejection sampling, n.: Generating many candidate answers and keeping only those that pass a check — a scoring model, a test or a correct final answer — often so the AI can then be trained on its own best work., reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do., reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse., test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter., value learningvalue learning, n.: Teaching an AI what people want by having it learn from their behavior, choices and feedback, rather than trying to write every goal down in advance., verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded., Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths., direct alignment algorithmdirect alignment algorithm, n.: Any method that tunes an AI directly on examples of preferred and dispreferred answers, skipping the separate scoring model and trial-and-error training used in classic RLHF.