preference data
In plain wordsRecords of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like.
Training data recording relative judgments about model outputs, most often triples of a prompt, a preferred ('chosen') response and a dispreferred ('rejected') one; richer formats include full rankings of several responses, per-attribute ratings, and unpaired binary labels of desirable or undesirable outputs. Preference data trains reward modelsreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. for reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. and is consumed directly by direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely. and related methods. It may come from paid annotators, from users (thumbs-up buttons, side-by-side comparisons) or from AI judges (reinforcement learning from AI feedbackreinforcement learning from AI feedback, n.: Training an AI with preference judgments made by another AI instead of by people, which is cheaper and faster but only as good as the AI doing the judging.). Important properties are whether responses were sampled from the current policy (on-policy) or from other models; inter-annotator agreementinter-annotator agreement, n.: A measure of how often different people labeling the same data give the same answer. Low agreement warns that the task is unclear or the labels are unreliable., which is often only around 60–75 percent on open-ended prompts; and the guidelines defining 'better', which encode contestable value choices. Public examples include Anthropic's HH-RLHF, Stanford's SHP and UltraFeedback.
We regenerated the preference data on-policy every week so the reward model kept up with the policy.
Origin
Paired comparison is an old method in psychometrics (Louis Thurstone's law of comparative judgment, 1927) and in sports rating. As a training resource for AI it dates from preference-based reinforcement learning (around 2011) and the 2017 work of Paul Christiano and colleagues; OpenAI's summarization comparisons (2020) and Anthropic's helpful-and-harmless dataset (2022) established today's chosen/rejected format.
Usage note
Preferences are relative, not absolute: a 'chosen' answer may still be bad. Because annotator pools, instructions and cultural backgrounds shape the labels, preference datasets are one channel through which particular values enter models.
Related
- reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.
- Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
- direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely.
- data labelingdata labeling, n.: The work of attaching correct answers to examples — often done by large paid workforces, sometimes by experts — so that AI systems can learn from them or be tested against them.
- inter-annotator agreementinter-annotator agreement, n.: A measure of how often different people labeling the same data give the same answer. Low agreement warns that the task is unclear or the labels are unreliable.
- LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings.
- instruction datasetinstruction dataset, n.: A collection of example requests paired with good answers, used to teach a pretrained language model to follow instructions and act as a helpful assistant.
Sources
- Bai, Y. et al. (2022). Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback. arXiv:2204.05862. arxiv.org/abs/2204.05862
- Stiennon, N. et al. (2020). Learning to summarize from human feedback. NeurIPS 2020. arXiv:2009.01325. arxiv.org/abs/2009.01325
Mentioned in
reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., training datatraining data, n.: The examples an AI system learns from. Whatever patterns, skills, gaps and biases are in the training data tend to show up in the finished system., Constitutional AIConstitutional AI, n.: Anthropic's technique for making an AI less harmful by having it check and rewrite its own answers against a short written set of principles, so that people need not label harmful examples by hand., direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely., instruction datasetinstruction dataset, n.: A collection of example requests paired with good answers, used to teach a pretrained language model to follow instructions and act as a helpful assistant., pretraining datapretraining data, n.: The enormous body of text, code and other material — mostly from the web — that a large AI model first learns from, before it is tuned to follow instructions or chat., reinforcement learning from AI feedbackreinforcement learning from AI feedback, n.: Training an AI with preference judgments made by another AI instead of by people, which is cheaper and faster but only as good as the AI doing the judging., reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples., self-improvementself-improvement, n.: When an AI gets better by learning from material it produces itself — its own answers, critiques or practice problems — rather than only from new human-made examples., Alexandr WangAlexandr Wang, prop. n.: An American entrepreneur who founded Scale AI, a major supplier of human-labeled data for training AI, and became Meta's chief AI officer in 2025., Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths., direct alignment algorithmdirect alignment algorithm, n.: Any method that tunes an AI directly on examples of preferred and dispreferred answers, skipping the separate scoring model and trial-and-error training used in classic RLHF., Kahneman–Tversky optimizationKahneman–Tversky optimization, n.: A way to tune an AI's behavior that needs only a thumbs-up or thumbs-down on single answers, rather than side-by-side comparisons, borrowing ideas from the psychology of how people weigh gains and losses., RewardBenchRewardBench, prop. n.: A public test that checks whether the scoring models used to train chatbots can tell a better answer from a worse one, by counting how often they prefer the right response.