Skip to content
Last Dictionary
Plane
Training & Post-trainingReinforcement LearningLanguage Models & Transformers
emerging

reinforcement learning with verifiable rewards

nounAlso: RLVR, verifiable rewards, verifiable reward, reinforcement learning from verifiable rewards

In plain wordsTraining an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.

  1. Reinforcement-learning fine-tuning of a language model in which the rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time. comes from a programmatic check of the output rather than from a learned reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.: exact match of a final math answer, passing a hidden test suite, a compiler or proof checker, or satisfaction of a format or instruction constraint. Because such rewards are hard to fool, policies can be optimized far longer than against learned rewards, and models trained this way learn to produce long chains of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate. with checking, backtracking and alternative strategies, the basis of reasoning modelsreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning.. Typical algorithms are proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback., group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average. and their variants, applied to a base or SFT model over thousands to millions of problems. Limits: rewards are usually sparse and binary, many valuable tasks have no checker, and imperfect checkers can still be gamed (reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.); rubric rewardsrubric reward, n.: A training score built from a checklist of specific things a good answer should do, with an AI grader ticking off each one — a way to reward quality where no single right answer exists. and process rewardsprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof. extend the idea beyond strictly verifiable domains.

    The RLVR run used 40,000 competition problems with integer answers, plus a sandbox that ran unit tests for code.

Origin

The name was introduced by Nathan Lambert and colleagues at the Allen Institute for AI in the Tülu 3 report (November 2024). Rewarding correct final answers had precedents including STaR (2022) and ReFT (2024); OpenAI's o1 (September 2024) showed the payoff of large-scale RL for reasoning, and DeepSeek-R1 (January 2025), trained with rule-based accuracy and format rewards, made the approach widely reproduced.

Usage note

Whether RLVR teaches new reasoning or mainly makes already-reachable solutions more likely is debated: some 2025 studies found base models match RL-trained ones when allowed many attempts (pass@kpass@k, n.: A score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code. at large k), while others report new capabilities after prolonged training. 'Verifiable' means automatically checkable, not formally proven.

Related

See also
  • reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning.
  • group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average.
  • verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded.
  • outcome reward modeloutcome reward model, n.: A model that judges only whether an AI's final answer is right, ignoring how it got there — like a teacher who checks just the answer box.
  • rubric rewardrubric reward, n.: A training score built from a checklist of specific things a good answer should do, with an AI grader ticking off each one — a way to reward quality where no single right answer exists.
  • reinforcement fine-tuningreinforcement fine-tuning, n.: Improving a trained AI model by trial and error instead of examples to copy: it tries answers, a grader scores them, and the model shifts toward whatever earned high scores.
  • DeepSeek-R1DeepSeek-R1, prop. n.: A free, downloadable AI reasoning model released by the Chinese company DeepSeek in January 2025. It matched leading American models at math and coding for a fraction of the reported cost, shaking the industry.
  • RL environmentRL environment, n.: A practice world built for training an AI by trial and error: it hands the AI tasks and tools, lets it act, and scores the results so the AI can learn what works.
Contrast with
  • reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.

Sources

  1. Lambert, N. et al. (2024). Tülu 3: Pushing Frontiers in Open Language Model Post-Training. arXiv:2411.15124. arxiv.org/abs/2411.15124
  2. DeepSeek-AI (2025). DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. arXiv:2501.12948 (revised version published in Nature, 2025). arxiv.org/abs/2501.12948
  3. Luong, T. Q. et al. (2024). ReFT: Reasoning with Reinforced Fine-Tuning. arXiv:2401.08967. arxiv.org/abs/2401.08967

Mentioned in

post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner., reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning., reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., rewardreward, n.: The score an AI system receives after an action, telling it how good the outcome was. Reinforcement learning trains the system to collect as much reward as possible over time., reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., synthetic datasynthetic data, n.: Data created by computer programs or AI models rather than collected from the real world — such as practice math problems written by a model, or simulated street scenes for self-driving cars., automated theorem provingautomated theorem proving, n.: Getting computers to prove mathematical or logical statements by themselves, by searching for a chain of valid steps that leads from basic assumptions to the statement., code generationcode generation, n.: Having an AI write computer code from a description in plain language, from finishing a single line as you type to building whole programs., DeepSeek-R1DeepSeek-R1, prop. n.: A free, downloadable AI reasoning model released by the Chinese company DeepSeek in January 2025. It matched leading American models at math and coding for a fraction of the reported cost, shaking the industry., exploration–exploitation trade-offexploration–exploitation trade-off, n.: The choice between trying something new that might turn out better and sticking with what already works — like picking a familiar restaurant versus testing a new one., group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average., LeanLean, prop. n.: A programming language and proof checker in which mathematicians write proofs that the computer verifies line by line. It has become the main language in which AI systems produce checkable proofs., mid-trainingmid-training, n.: A middle stage in building a language model, between its broad initial training and its tuning into an assistant: it keeps learning the same way, but from a smaller, carefully chosen diet of math, code, reasoning or long documents., OpenAI o1OpenAI o1, prop. n.: OpenAI's first reasoning model, from 2024. Before replying it writes out a long, hidden series of steps, trying ideas and checking them, which lifted its scores on hard math, science and coding problems far above earlier chatbots'., reasoningreasoning, n.: Working out conclusions step by step from facts, rules or evidence, as in solving a math problem or planning a trip. Whether today's AI truly reasons, or imitates reasoning, is debated., reinforcement fine-tuningreinforcement fine-tuning, n.: Improving a trained AI model by trial and error instead of examples to copy: it tries answers, a grader scores them, and the model shifts toward whatever earned high scores., RL environmentRL environment, n.: A practice world built for training an AI by trial and error: it hands the AI tasks and tools, lets it act, and scores the results so the AI can learn what works., RL scalingRL scaling, n.: The practice, and the finding behind it, of giving AI models far more trial-and-error practice after their initial training, especially on problems with checkable answers, which has so far improved their reasoning fairly predictably., sparse rewardsparse reward, n.: A situation in which an AI agent gets feedback only rarely — say, only when it finally wins — so it has almost nothing to learn from until it stumbles on success., verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded., agentic reinforcement learningagentic reinforcement learning, n.: Training an AI assistant by rewarding success on long, multi-step jobs where it must use tools, browse, run code and react to what happens, rather than answer in one go., autoformalizationautoformalization, n.: Using AI to translate ordinary mathematics written for people — definitions, theorem statements, proofs — into a strict formal language that a proof-checking program can verify., entropy collapseentropy collapse, n.: A failure in reinforcement-learning training where an AI quickly becomes overconfident and stops trying different answers, so it explores less and its improvement stalls.