entropy collapse
In plain wordsA failure in reinforcement-learning training where an AI quickly becomes overconfident and stops trying different answers, so it explores less and its improvement stalls.
A rapid fall in the entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy. of a policy's output distribution during reinforcement-learning training, so that the model becomes nearly deterministic, produces near-identical samples and stops exploring. In RL fine-tuning of language models, especially reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds., entropy often drops sharply in the first steps while accuracy plateaus; when every sample in a group is the same, methods such as group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average. receive little learning signal. A 2025 analysis found performance following an approximate exponential law in entropy, implying a ceiling once entropy is spent, and traced the decline to the covariance between a token's probability and its advantageadvantage function, n.: A measure of how much better or worse one particular move is than the agent's typical move in the same situation. Positive means better than usual; negative means worse. under policy-gradient updates. Remedies include an entropy regularizationentropy regularization, n.: Rewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early. bonus, raising the upper clipping bound of the importance ratio (as in DAPO), restricting updates on high-covariance tokens, higher sampling temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone., and filtering out prompts the model always solves or always fails.
Entropy collapsed within 200 steps, and after that every rollout in a group was the same answer.
Origin
Premature convergence and entropy loss are long-standing concerns in policy-gradient RL, which is why entropy bonuses were added to methods such as A3C (2016). The phrase became prominent in 2025 work on reasoning RL, notably DAPO (March 2025) and Ganqu Cui and colleagues' May 2025 study of the entropy mechanism of RL for reasoning language models.
Usage note
Not to be confused with mode collapsemode collapse, n.: When a generator gets stuck producing only a few kinds of output — the same face, the same joke — instead of the full variety it was meant to learn. in generative adversarial networks or model collapsemodel collapse, n.: The gradual decline of AI models trained on the output of earlier AI models: rare details vanish first, then quality and variety erode, like a photocopy of a photocopy. from training on synthetic data, though all three describe a loss of output diversity.
Related
- entropy regularizationentropy regularization, n.: Rewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early.
- exploration–exploitation trade-offexploration–exploitation trade-off, n.: The choice between trying something new that might turn out better and sticking with what already works — like picking a familiar restaurant versus testing a new one.
- group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average.
- reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.
- entropyentropy, n.: A measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy.
- temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone.
- mode collapsemode collapse, n.: When a generator gets stuck producing only a few kinds of output — the same face, the same joke — instead of the full variety it was meant to learn.
- model collapsemodel collapse, n.: The gradual decline of AI models trained on the output of earlier AI models: rare details vanish first, then quality and variety erode, like a photocopy of a photocopy.
Sources
- Cui, G. et al. (2025). The Entropy Mechanism of Reinforcement Learning for Reasoning Language Models. arXiv:2505.22617. arxiv.org/abs/2505.22617
- Yu, Q. et al. (2025). DAPO: An Open-Source LLM Reinforcement Learning System at Scale. arXiv:2503.14476. arxiv.org/abs/2503.14476