reinforcement learning from human feedback
In plain wordsA way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
A method for optimizing a model toward behavior people prefer when that behavior is hard to specify as a reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do.. In the canonical three-stage recipe for language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.: (1) supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples. on demonstrations; (2) collection of preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like., in which people compare or rank several responses to the same prompt, and training of a reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. on it, usually with a Bradley–TerryBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths. loss; (3) reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., typically proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback., to maximize the learned reward minus a KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer. that keeps the policy near a reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started.. RLHF made models markedly more helpful and less toxic at modest cost and underpinned ChatGPTChatGPT, prop. n.: OpenAI's AI chatbot, released free to the public in November 2022, which answers questions, writes and converses in everyday language. Its sudden popularity began the current AI boom.. Its weaknesses include expensive, noisy labels, reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse. and reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric., sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful., and the difficulty of supervising tasks people cannot evaluate, which motivates scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.. Offline alternatives such as direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely. target the same objective without an explicit reward model or RL loop.
Our RLHF run used 300,000 pairwise comparisons, and the reward model was half the size of the policy.
Origin
Agents that learn from human evaluative feedback were studied in the 2000s, for example TAMER (W. Bradley Knox and Peter Stone, 2008–2009), as was preference-based reinforcement learning (Riad Akrour and colleagues; Weiwei Cheng and colleagues, 2011). Paul Christiano and colleagues at OpenAI and DeepMind scaled it to deep RL in 2017 with pairwise preferences over Atari and simulated-robot behavior; Daniel Ziegler (2019) and Nisan Stiennon (2020) and colleagues applied it to language, and InstructGPT and ChatGPT (2022) made 'RLHF' a household acronym.
Usage note
Usage varies: narrowly, RLHF means reward model plus RL; broadly, any preference-based post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner., including DPO. Early papers said 'learning from human preferences'. Critics note that RLHF optimizes for what raters approve of, which can diverge from what is true or good.
Related
- reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.
- preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like.
- proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.
- KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer.
- direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely.
- reinforcement learning from AI feedbackreinforcement learning from AI feedback, n.: Training an AI with preference judgments made by another AI instead of by people, which is cheaper and faster but only as good as the AI doing the judging.
- InstructGPTInstructGPT, prop. n.: A 2022 OpenAI language model trained with human feedback to follow instructions; people preferred its answers to those of a model 100 times larger, and its recipe led directly to ChatGPT.
- reward hackingreward hacking, n.: When an AI finds a shortcut that earns the reward or score it is trained on without doing the task its designers wanted — like a student gaming the grading rubric.
- supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples.
- reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.
Sources
- Christiano, P., Leike, J., Brown, T. B., Martic, M., Legg, S. & Amodei, D. (2017). Deep reinforcement learning from human preferences. NeurIPS 2017. arXiv:1706.03741. arxiv.org/abs/1706.03741
- Stiennon, N. et al. (2020). Learning to summarize from human feedback. NeurIPS 2020. arXiv:2009.01325. arxiv.org/abs/2009.01325
- Ouyang, L. et al. (2022). Training language models to follow instructions with human feedback. NeurIPS 2022. arXiv:2203.02155. arxiv.org/abs/2203.02155
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., ChatGPTChatGPT, prop. n.: OpenAI's AI chatbot, released free to the public in November 2022, which answers questions, writes and converses in everyday language. Its sudden popularity began the current AI boom., ClaudeClaude, prop. n.: A family of AI assistants and language models made by the company Anthropic since 2023, widely used for writing, analysis and computer programming., Dario AmodeiDario Amodei, prop. n.: An American AI researcher who co-founded Anthropic and is its chief executive, known for early work on scaling and AI safety and for forecasting very rapid AI progress., fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way., large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., OpenAIOpenAI, prop. n.: The American company that makes ChatGPT and the GPT models. It began in 2015 as a nonprofit research lab and is now a for-profit company overseen by a nonprofit foundation., post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner., pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., refusalrefusal, n.: When an AI declines to do what it was asked — 'I can't help with that' — usually because the request seems harmful or breaks the rules it was trained to follow., reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time., sycophancysycophancy, n.: When an AI tells people what they want to hear — flattering them, agreeing with their mistakes or dropping a correct answer under pushback — instead of what is true or useful., trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better., AI assistantAI assistant, n.: An AI program, such as ChatGPT, Claude or Gemini, that helps a person through conversation: answering questions, writing, summarizing, coding and, increasingly, taking actions on their behalf., annotationannotation, n.: Adding information to raw data by hand or with software — marking objects in photos, tagging names in sentences, transcribing speech or ranking chatbot answers — so machines can learn from it., chat modelchat model, n.: A language model trained to hold a back-and-forth conversation, keeping track of who said what and replying as a helpful assistant. It is the engine inside chat apps., Constitutional AIConstitutional AI, n.: Anthropic's technique for making an AI less harmful by having it check and rewrite its own answers against a short written set of principles, so that people need not label harmful examples by hand., delvedelve, v.: An ordinary English verb meaning to dig into a subject, which AI chatbots used so often in 2023–2024 that it became a joke and a supposed telltale sign of machine-written text., direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely., feedback loopfeedback loop, n.: A cycle in which what a system produces comes back to shape its own future input, like a microphone howling near its loudspeaker. In AI, a model's predictions can shape the very data it later learns from., ghost workghost work, n.: The hidden human labor behind AI: people, often poorly paid and far away, who label data, check outputs and handle cases the software cannot, while the product appears fully automatic., helpful, honest and harmlesshelpful, honest and harmless, phr.: A three-part goal for AI assistants, popularized by Anthropic: be genuinely useful, tell the truth, and avoid causing harm — shorthand for what an aligned chatbot should be., human evaluationhuman evaluation, n.: Judging an AI's outputs by asking people to rate or compare them, because for many tasks — writing, conversation, translation — no automatic score captures quality well., human-in-the-loophuman-in-the-loop, adj.: Describes AI systems in which a person checks, corrects or approves the machine's work at key points, rather than letting it act entirely on its own.