outcome reward model
In plain wordsA model that judges only whether an AI's final answer is right, ignoring how it got there — like a teacher who checks just the answer box.
A reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. trained only on whether complete solutions reach a correct final result, which it learns to predict from the full text of a solution (often at every token, so it can also estimate the chance of success partway through). ORMs need only answer-level labels, which are cheap whenever a reference answer exists, and are used to rerank samples in best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers., to filter self-generated training data, and as rewards in reinforcement learning. Their weakness is that a solution can reach the right answer through flawed reasoning, or the wrong answer after mostly sound steps, and the ORM cannot say where things went wrong; the step-level alternative is a process reward modelprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof.. When answers can be checked by rule, a learned ORM is often replaced by an exact verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded., as in reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds..
Reranking 100 samples with the ORM lifted accuracy by nine points, yet some top-ranked solutions had broken reasoning.
Origin
Karl Cobbe and colleagues at OpenAI (2021) trained such models, which they called verifiers, to judge solutions to GSM8K math problems. Jonathan Uesato and colleagues at DeepMind (2022) introduced the contrast between outcome-supervised and process-supervised reward models and the abbreviations ORM and PRM, which Hunter Lightman and colleagues (2023) adopted.
Usage note
Rewards from a rule-based checker, such as exact answer match, are not ORMs in the strict sense, since nothing is learned, though some writers use the term loosely for any outcome-level reward.
Related
- process reward modelprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof.
- verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded.
- reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.
- best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers.
- reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.
- GSM8KGSM8K, prop. n.: A set of about 8,500 grade-school math word problems, each needing a few steps of simple arithmetic, used to test whether language models can reason through a problem step by step.
- process reward modelprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof.
Sources
- Cobbe, K. et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168. arxiv.org/abs/2110.14168
- Uesato, J. et al. (2022). Solving math word problems with process- and outcome-based feedback. arXiv:2211.14275. arxiv.org/abs/2211.14275
Mentioned in
reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds., reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples., verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded., outcome supervisionoutcome supervision, n.: Training an AI by rewarding it only for whether the end result is right — the answer, the passing test, the won game — without grading how it got there., process reward modelprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof., process supervisionprocess supervision, n.: Training an AI by grading each step of its reasoning or work, not just whether the final answer came out right, so that it learns sound methods instead of lucky shortcuts.