self-correction
In plain wordsAn AI model noticing and fixing its own mistakes — by rereading its answer, running a test or checking a source — before it gives a final result.
The process by which a model revises its own output after detecting errors, within one response or across turns of an agent loopagent loop, n.: The repeating cycle at the heart of an AI agent: the model decides what to do next, software carries it out, the result is shown to the model, and the cycle repeats until the task is finished.. Researchers distinguish intrinsic self-correction, in which the model critiques and revises its answer using only its own judgment, from extrinsic self-correction, which uses outside signals such as compiler errors, failing tests, tool results, retrieved evidence or a separate verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded.. Evidence favors the second: models fix errors reliably when feedback is informative, but studies found that simply prompting a model to review its answer often fails to improve, and sometimes worsens, reasoning accuracy. Training can instill the skill — for instance multi-turn reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. on the model's own correction attempts — and reasoning modelsreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning. learn to backtrack within their chains of thought. For agents, noticing and recovering from mistakes largely determines success on long-horizon taskslong-horizon task, n.: A job that takes many steps and a long time — hours or days of work rather than a single answer — so an AI must stay on track and recover from its own mistakes..
Self-correction kicked in when the build failed: the agent reread the error, fixed the import and reran the tests.
Origin
Generate–critique–revise methods for language models appeared in 2023, notably Self-Refine (Aman Madaan et al., March 2023), ReflexionReflexion, prop. n.: A technique in which an AI agent that fails at a task writes itself a short note on what went wrong, then reads that note before trying again — learning from mistakes without retraining. (Noah Shinn et al., March 2023) and self-debugging for code (Xinyun Chen et al., April 2023). Jie Huang and colleagues' 'Large Language Models Cannot Self-Correct Reasoning Yet' (October 2023; ICLR 2024) prompted the intrinsic–extrinsic distinction; Google DeepMind's SCoRe (2024) trained self-correction with reinforcement learning.
Usage note
Claims of self-correction should state what feedback was available: gains measured with oracle labels — stopping once the answer is known to be right — overstate real-world ability.
Related
- ReflexionReflexion, prop. n.: A technique in which an AI agent that fails at a task writes itself a short note on what went wrong, then reads that note before trying again — learning from mistakes without retraining.
- evaluator–optimizerevaluator–optimizer, n.: A pairing in which one AI produces a draft and another AI grades it and suggests fixes, repeating the cycle until the work is good enough.
- verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded.
- reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning.
- long-horizon tasklong-horizon task, n.: A job that takes many steps and a long time — hours or days of work rather than a single answer — so an AI must stay on track and recover from its own mistakes.
- chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate.
Sources
- Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X. & Zhou, D. (2023). Large Language Models Cannot Self-Correct Reasoning Yet. arXiv:2310.01798 (ICLR 2024). arxiv.org/abs/2310.01798
- Kumar, A. et al. (2024). Training Language Models to Self-Correct via Reinforcement Learning. arXiv:2409.12917. arxiv.org/abs/2409.12917
- Madaan, A. et al. (2023). Self-Refine: Iterative Refinement with Self-Feedback. arXiv:2303.17651. arxiv.org/abs/2303.17651
Mentioned in
long-horizon tasklong-horizon task, n.: A job that takes many steps and a long time — hours or days of work rather than a single answer — so an AI must stay on track and recover from its own mistakes., evaluator–optimizerevaluator–optimizer, n.: A pairing in which one AI produces a draft and another AI grades it and suggests fixes, repeating the cycle until the work is good enough., ReflexionReflexion, prop. n.: A technique in which an AI agent that fails at a task writes itself a short note on what went wrong, then reads that note before trying again — learning from mistakes without retraining.