process supervision
In plain wordsTraining an AI by grading each step of its reasoning or work, not just whether the final answer came out right, so that it learns sound methods instead of lucky shortcuts.
Training in which a model is graded on the soundness of each step of its reasoning or work, not only on where it ends up. In mathematical reasoning, humans or models label each step of a solution as correct or flawed, and a process reward modelprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof. trained on these labels scores new solutions step by step for reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. or best-of-nbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers. reranking. Proponents see safety advantages: it rewards reasoning humans endorse, localizes errors and avoids reinforcing correct answers reached by flawed or deceptive means. Its costs are expensive step-level labels and the risk that models learn steps that merely look good to graders. Since 2024, the success of reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds., which rewards outcomes only, has reopened the question of how much process supervision is needed.
With process supervision, the reward model penalized the algebra slip in step four even though the final answer was right.
Origin
Advocated as an alignment strategy by Andreas Stuhlmüller and Jungwon Byun of Ought in 'Supervise Process, not Outcomes' (2022). Jonathan Uesato and colleagues at DeepMind (2022) compared process- and outcome-based feedback on grade-school math, and Hunter Lightman and colleagues at OpenAI ('Let's Verify Step by Step', 2023) released about 800,000 step-level human labels (PRM800K).
Usage note
In the safety literature, 'process-based' sometimes means more broadly that the training signal should depend on whether an overseer understands and endorses the steps taken, not on their downstream consequences.
Related
- process reward modelprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof.
- outcome supervisionoutcome supervision, n.: Training an AI by rewarding it only for whether the end result is right — the answer, the passing test, the won game — without grading how it got there.
- outcome reward modeloutcome reward model, n.: A model that judges only whether an AI's final answer is right, ignoring how it got there — like a teacher who checks just the answer box.
- chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate.
- reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
- scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
- outcome supervisionoutcome supervision, n.: Training an AI by rewarding it only for whether the end result is right — the answer, the passing test, the won game — without grading how it got there.
Sources
- Lightman, H. et al. (2023). Let's Verify Step by Step. arXiv preprint; ICLR 2024.
- Uesato, J. et al. (2022). Solving math word problems with process- and outcome-based feedback. arXiv preprint.
Mentioned in
myopiamyopia, n.: Designing an AI to care only about the immediate results of its actions, not about what happens far in the future, so it has no reason for long-term scheming., outcome supervisionoutcome supervision, n.: Training an AI by rewarding it only for whether the end result is right — the answer, the passing test, the won game — without grading how it got there., process reward modelprocess reward model, n.: A model that grades each step of an AI's worked solution, not just the final answer, so mistakes can be caught where they happen — like a teacher marking every line of a proof.