generative reward model
In plain wordsA scoring model that judges an AI's answer by writing out its reasoning in words before giving a verdict, instead of simply outputting a number.
A reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. or verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded. implemented as an ordinary text-generating language model that evaluates a response by producing tokens — a critique, a chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate., then a verdict or score — rather than through a scalar output head; the reward is read from the verdict or from the probability the model assigns to a 'correct' token. Being an ordinary language model at heart (sometimes trained to write solutions as well as to judge them), it can draw on the base model's reasoning, follow rubrics or principles stated in its instructions, and become more reliable on hard cases by spending extra test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter., for example by sampling several critiques and going with the majority verdict. Compared with discriminative (classifier-head) reward models it costs more per judgment and its written rationale may not reflect how it actually decided, but it has outperformed them at verifying reasoning. It is the trained counterpart of prompted LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers. evaluation.
Our generative reward model writes a short critique of each proof before scoring it, which makes its mistakes easy to audit.
Origin
Prompted model judges spread in 2023 under the name LLM-as-a-judge. Lunjun Zhang and colleagues (August 2024) introduced generative verifiers (GenRM) trained by next-token prediction, raising best-of-n accuracy on GSM8K from 73 to 93.4 percent; Dakota Mahan and colleagues' Generative Reward Models (October 2024) and later work extended the approach to general preferences.
Usage note
Terminology is unsettled: 'GenRM' names specific methods in at least two 2024 papers, and 'critic model' is also used. The line between this and LLM-as-a-judge is mainly whether the judge was trained for the role.
Related
- reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.
- verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded.
- LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.
- process reward modelprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof.
- rubric rewardrubric reward, n.: A training score built from a checklist of specific things a good answer should do, with an AI grader ticking off each one — a way to reward quality where no single right answer exists.
- reinforcement learning from AI feedbackreinforcement learning from AI feedback, n.: Training an AI with preference judgments made by another AI instead of by people, which is cheaper and faster but only as good as the AI doing the judging.
Sources
- Zhang, L., Hosseini, A., Bansal, H., Kazemi, M., Kumar, A. & Agarwal, R. (2024). Generative Verifiers: Reward Modeling as Next-Token Prediction. arXiv:2408.15240. arxiv.org/abs/2408.15240
- Mahan, D. et al. (2024). Generative Reward Models. arXiv preprint.
Mentioned in
RewardBenchRewardBench, prop. n.: A public test that checks whether the scoring models used to train chatbots can tell a better answer from a worse one, by counting how often they prefer the right response.