recursive reward modeling
In plain wordsA plan for supervising advanced AI in stages: AI assistants trained on human feedback help people judge the work of a more capable AI, which then helps judge the next one.
A scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking. proposal that extends reward modeling — learning a reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. from human feedback and training an agent against it — by recursion. Agents already trained this way help a human evaluate the outcomes of a harder task, for instance by summarizing, critiquing or checking parts of it, and these assisted judgments train the reward model for the next agent. Each level requires only that evaluating the task, with help, is easier than performing it. The approach shares its decomposition logic with iterated amplificationiterated amplification, n.: A proposed training scheme in which a person, helped by many copies of an AI, solves harder problems than either could alone; a new AI then learns to imitate that team, and the cycle repeats. but supervises outcomes through learned rewards rather than imitating an amplified process. Like it, it risks compounding errors across levels if an assistant is subtly wrong or misaligned. Model-written critiques that help raters find flaws are a practical descendant.
In recursive reward modeling, the critique assistant from the last round helps raters score the next agent's proofs.
Origin
Proposed by Jan Leike, David Krueger, Tom Everitt, Miljan Martic, Vishal Maini and Shane Legg of DeepMind in 'Scalable agent alignment via reward modeling: a research direction' (2018). OpenAI's recursive book-summarization (2021) and self-critique (2022) experiments were framed as early steps in this direction.
Related
- reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
- scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
- iterated amplificationiterated amplification, n.: A proposed training scheme in which a person, helped by many copies of an AI, solves harder problems than either could alone; a new AI then learns to imitate that team, and the cycle repeats.
- AI debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend.
- reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
Sources
- Leike, J., Krueger, D., Everitt, T., Martic, M., Maini, V. & Legg, S. (2018). Scalable agent alignment via reward modeling: a research direction. arXiv:1811.07871.
Mentioned in
scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking., AI debateAI debate, n.: A proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend., iterated amplificationiterated amplification, n.: A proposed training scheme in which a person, helped by many copies of an AI, solves harder problems than either could alone; a new AI then learns to imitate that team, and the cycle repeats., Jan LeikeJan Leike, prop. n.: A German AI-alignment researcher who helped develop training from human feedback, co-led OpenAI's superalignment team, and now leads alignment research at Anthropic.