RewardBench
In plain wordsA public test that checks whether the scoring models used to train chatbots can tell a better answer from a worse one, by counting how often they prefer the right response.
A benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. for reward modelsreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples., the learned scorers used in reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. and in reranking outputs. Each item pairs a prompt with a preferred ('chosen') and a dispreferred ('rejected') response, and a model is credited when it scores the chosen one higher; implicit rewards of DPOdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely.-trained models and generative judgesLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers. can be scored the same way. The 2024 release grouped about 3,000 curated comparisons into Chat, Chat Hard, Safety and Reasoning subsets, many hinging on one concrete flaw, such as a planted code bug or an ignored instruction. As leading models neared the ceiling, RewardBench 2 (2025) introduced new human-written prompts and a harder best-of-four format, in which the correct response must outscore three wrong ones, across domains including factuality, precise instruction following, math and safety; scores fell about 20 points, and its authors find that results on it track a reward model's usefulness for best-of-n rerankingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers. and examine how they bear on RLHF training.
The new reward model topped RewardBench but made our PPO run worse, so we stopped relying on the leaderboard alone.
Origin
Released in March 2024 by Nathan Lambert and colleagues at the Allen Institute for AIAllen Institute for AI, prop. n.: Paul Allen, a co-founder of Microsoft, created this nonprofit Seattle AI lab in 2014. It is known for releasing models together with everything used to build them, and for the Semantic Scholar paper search engine. (Ai2) and collaborating institutions. RewardBench 2 followed in June 2025 from Saumya Malik, Valentina Pyatkin, Lambert and co-authors, and was presented at ICLR 2026.
Usage note
High accuracy on fixed preference pairs does not guarantee that a reward model resists reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric. or reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse. once a policy is optimized against it. Weak correlation with downstream training results was the main critique of the first version and the main design goal of the second.
Related
- reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
- preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like.
- generative reward modelgenerative reward model, n.: A scoring model that judges an AI's answer by writing out its reasoning in words before giving a verdict, instead of simply outputting a number.
- reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse.
- best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
Sources
- Lambert, N. et al. (2024). RewardBench: Evaluating Reward Models for Language Modeling. arXiv preprint.
- Malik, S., Pyatkin, V., Land, S., Morrison, J., Smith, N. A., Hajishirzi, H. & Lambert, N. (2025). RewardBench 2: Advancing Reward Model Evaluation. arXiv:2506.01937; ICLR 2026. arxiv.org/abs/2506.01937v1