reinforcement learning from AI feedback
In plain wordsTraining an AI with preference judgments made by another AI instead of by people, which is cheaper and faster but only as good as the AI doing the judging.
A variant of reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. in which the preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like., or the reward itself, comes from a language model rather than human annotators. Typically a prompted judge model compares two responses against written criteria, such as the principles of an AI constitutionAI constitution, n.: A written set of principles that an AI company uses to shape an AI's values and behavior, both during training and as a public statement of what the AI should be like., and its judgments train a reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. that drives reinforcement learning; in direct RLAIF the judge scores outputs during RL with no separate reward model. AI feedback scales to millions of labels at a small fraction of human cost and makes the criteria explicit and editable, but it imports the judge's biases (for example toward longer or stylistically familiar answers) and risks circularity when a model grades its own family. It is a component of Constitutional AIConstitutional AI, n.: Anthropic's technique for making an AI less harmful by having it check and rewrite its own answers against a short written set of principles, so that people need not label harmful examples by hand. and of most modern post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner. pipelines, alongside LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers. evaluation.
For harmlessness we used RLAIF, with a judge model applying the constitution to 400,000 response pairs.
Origin
Named in Anthropic's Constitutional AI paper (Yuntao Bai and colleagues, December 2022), which called its reinforcement-learning phase 'RL from AI Feedback (RLAIF)'. Harrison Lee and colleagues at Google (2023) found RLAIF on a par with RLHF for summarization and dialogue and introduced direct RLAIF.
Usage note
RLAIF does not remove people: they still write the principles, choose the judge and audit results. Related ideas include scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking. and self-improvementself-improvement, n.: When an AI gets better by learning from material it produces itself — its own answers, critiques or practice problems — rather than only from new human-made examples..
Related
- Constitutional AIConstitutional AI, n.: Anthropic's technique for making an AI less harmful by having it check and rewrite its own answers against a short written set of principles, so that people need not label harmful examples by hand.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
- LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers.
- reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.
- AI constitutionAI constitution, n.: A written set of principles that an AI company uses to shape an AI's values and behavior, both during training and as a public statement of what the AI should be like.
- scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
- self-improvementself-improvement, n.: When an AI gets better by learning from material it produces itself — its own answers, critiques or practice problems — rather than only from new human-made examples.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
Sources
- Bai, Y. et al. (2022). Constitutional AI: Harmlessness from AI Feedback. arXiv:2212.08073. arxiv.org/abs/2212.08073
- Lee, H. et al. (2023). RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback. arXiv:2309.00267. arxiv.org/abs/2309.00267
Mentioned in
post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner., reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., Constitutional AIConstitutional AI, n.: Anthropic's technique for making an AI less harmful by having it check and rewrite its own answers against a short written set of principles, so that people need not label harmful examples by hand., LLM-as-a-judgeLLM-as-a-judge, n.: Using one AI language model to grade or compare the answers of another, as a cheaper and faster stand-in for human reviewers., model specificationmodel specification, n.: A public document in which an AI company spells out how its models are supposed to behave — what they should do, refuse and prioritize, and whom to obey when instructions conflict., preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like., recursive self-improvementrecursive self-improvement, n.: An AI improving its own design or training, so that each improved version is better at making the next improvement. It is the engine usually imagined behind an intelligence explosion., safety trainingsafety training, n.: The part of an AI's training that teaches it to avoid harmful behavior — declining dangerous requests, resisting manipulation and following its developer's rules — on top of being generally helpful., self-improvementself-improvement, n.: When an AI gets better by learning from material it produces itself — its own answers, critiques or practice problems — rather than only from new human-made examples., generative reward modelgenerative reward model, n.: A scoring model that judges an AI's answer by writing out its reasoning in words before giving a verdict, instead of simply outputting a number., rubric rewardrubric reward, n.: A training score built from a checklist of specific things a good answer should do, with an AI grader ticking off each one — a way to reward quality where no single right answer exists.