Skip to content
Last Dictionary
Plane
Training & Post-trainingLanguage Models & TransformersReinforcement Learning
emerging

on-policy distillation

nounAlso: generalized knowledge distillation, GKD, on-policy KD

In plain wordsA way of teaching a small AI from a bigger one where the small AI writes its own answers and the bigger one grades every word, so the learner is corrected on its own mistakes.

  1. Knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run. in which the student is trained on sequences it generates itself, with the teacher supplying a target probability distribution at every token of those sequences. Training minimizes a per-token divergence between student and teacher — commonly the reverse Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right., which pushes the student toward the teacher's main modes — over the states the student actually visits. Like reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., it trains on the student's own outputs, so the student learns to recover from its own mistakes and avoids the train–inference mismatch of teacher-forcedteacher forcing, n.: Training a model that writes one word at a time by always feeding it the correct previous words, rather than its own possibly mistaken guesses, while it learns to predict the next one. distillation; like supervised learning, it receives a signal at every token rather than one score per answer; and it needs no reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.. It can be read as RL in which each token's reward is the teacher's log-probability minus the student's. Reported uses include training small reasoning models at a fraction of the compute of RL, and restoring behaviors such as instruction following that were lost during domain fine-tuning.

    On-policy distillation from the 32B teacher matched our RL run's math scores with a tenth of the GPU-hours.

L(θ)=Ex,  y∼πθ(⋅∣x)∑tKL(πθ(⋅∣x,y<t) ∥ πT(⋅∣x,y<t))\mathcal{L}(\theta) = \mathbb{E}_{x,\; y \sim \pi_\theta(\cdot \mid x)} \sum_{t} \mathrm{KL}\big(\pi_\theta(\cdot \mid x, y_{<t}) \,\|\, \pi_{\mathrm{T}}(\cdot \mid x, y_{<t})\big)

Origin

Training on the learner's own trajectories under expert correction comes from imitation learning (DAgger, by Stéphane Ross, Geoffrey Gordon and J. Andrew Bagnell, 2011). Rishabh Agarwal and colleagues at Google DeepMind introduced generalized knowledge distillation for language models in June 2023, and MiniLLM (Yuxian Gu and colleagues, 2023) used reverse KL. The term spread in 2025 through Qwen3's small models and an October 2025 Thinking Machines Lab post by Kevin Lu.

Usage note

It needs the teacher's token probabilities and usually a shared tokenizer, so it is impractical with closed models that return only text. Contrast off-policy distillation, which fine-tunes on teacher-written text.

Related

See also
  • knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.
  • on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes.
  • imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error.
  • teacher forcingteacher forcing, n.: Training a model that writes one word at a time by always feeding it the correct previous words, rather than its own possibly mistaken guesses, while it learns to predict the next one.
  • reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.
  • Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right.
  • catastrophic forgettingcatastrophic forgetting, n.: When a neural network trained on something new gets suddenly and badly worse at what it could already do, because the new training rewrites the internal settings the old skills relied on.
Contrast with
  • supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples.

Sources

  1. Agarwal, R. et al. (2023). On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes. ICLR 2024. arXiv:2306.13649. arxiv.org/abs/2306.13649
  2. Lu, K. (2025). On-Policy Distillation. Thinking Machines Lab blog, 27 October 2025. thinkingmachines.ai/blog/on-policy-distillation

Mentioned in

knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run., catastrophic forgettingcatastrophic forgetting, n.: When a neural network trained on something new gets suddenly and badly worse at what it could already do, because the new training rewrites the internal settings the old skills relied on., on-policyon-policy, adj.: Describes learning that uses only experience produced by the strategy currently being trained, so old data must be thrown away each time the strategy changes., DAggerDAgger, n.: A way of teaching a robot by imitation in which an expert keeps labeling what it should have done in the situations it actually gets into, so it learns to recover from its own mistakes., exposure biasexposure bias, n.: A weakness of models trained only on perfect examples: when they write on their own, one early mistake puts them in unfamiliar territory they never practiced, and errors can snowball.