Skip to content
Last Dictionary
Plane
Training & Post-trainingReinforcement LearningLanguage Models & TransformersAGI & Superintelligence
emergingcontested

RL scaling

nounAlso: reinforcement learning scaling, RL compute scaling, scaling RL, RL scaling laws

In plain wordsThe practice, and the finding behind it, of giving AI models far more trial-and-error practice after their initial training, especially on problems with checkable answers, which has so far improved their reasoning fairly predictably.

  1. Increasing the compute spent on reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. in the post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner. of language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., and the empirical regularities linking that compute to capability. In the typical setup, a model writes long chains of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate. for problems whose outcomes can be checked automatically (math answers, unit tests, other verifiers) and is updated with policy-gradient methods such as PPOproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback. or GRPOgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average. under verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.; performance rises with RL training compute, and the resulting reasoning modelsreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning. also improve with test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter.. Laboratories have reported steep increases: OpenAI said in April 2025 that it had pushed RL training compute a further order of magnitude for o3, and xAI said that the RL for Grok 4 (July 2025) ran at pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses. scale. Systematic studies fit sigmoid-shaped curves of performance against RL compute and find that most recipe choices change efficiency while a few change the ceiling. Open questions include how far gains transfer beyond verifiable domains, whether RL teaches new abilities or mainly sharpens latent ones, and how reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric. grows with scale.

    Most of the jump from the previous release came from RL scaling, not from a bigger base model.

Origin

OpenAI's o1OpenAI o1, prop. n.: OpenAI's first reasoning model, from 2024. Before replying it writes out a long, hidden series of steps, trying ideas and checking them, which lifted its scores on hard math, science and coding problems far above earlier chatbots'. announcement (September 2024) showed accuracy rising smoothly with both RL training compute and thinking time, and DeepSeek-R1 (January 2025) published an open recipe. The phrase spread through 2025 as laboratories described RL as a new scaling axis; Devvrit Khatri and colleagues' October 2025 study, using more than 400,000 GPU-hours, fitted predictive RL compute curves.

Usage note

Contested in two ways. Some studies find RL-trained models solve fewer distinct problems than their base models when many attempts are allowed (high-k pass@kpass@k, n.: A score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code.), suggesting elicitation rather than new capability, while others report that prolonged RL widens the solvable set. And a scalar reward per long episode carries far less training signal than pretraining text, raising doubts about efficiency at scale. Curves depend on recipe and task.

Related

See also
  • scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
  • reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.
  • reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning.
  • test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter.
  • post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner.
  • group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average.
  • DeepSeek-R1DeepSeek-R1, prop. n.: A free, downloadable AI reasoning model released by the Chinese company DeepSeek in January 2025. It matched leading American models at math and coding for a fraction of the reported cost, shaking the industry.
  • capability elicitationcapability elicitation, n.: The work of drawing out the best an AI can do on a test — with better instructions, tools, retries or extra training — so that its abilities are not underestimated.
Contrast with
  • pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.

Sources

  1. OpenAI (2024). Learning to Reason with LLMs. OpenAI blog, 12 September 2024.
  2. Khatri, D. et al. (2025). The Art of Scaling Reinforcement Learning Compute for LLMs. arXiv:2510.13786. arxiv.org/abs/2510.13786v1
  3. Yue, Y. et al. (2025). Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model? arXiv preprint.