Skip to content
Last Dictionary
Plane
Training & Post-trainingLanguage Models & TransformersReinforcement Learning

direct preference optimization

nounAlso: DPO

In plain wordsA simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely.

  1. An offline method for aligning a language model with preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like. that optimizes the same KL-regularized objective as reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. without training a separate reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. or running reinforcement learning. Its key observation is that the optimal policy under a KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer. is a reweighting of the reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started., so the reward can be rewritten as β times the log-ratio of policy to reference probabilities; substituting this into the Bradley–TerryBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths. likelihood turns preference learning into a logistic loss on (prompt, chosen, rejected) triples. Needing no sampling during training and few hyperparameters, DPO is cheap and stable, and it became the default preference-tuning method for open-weight models from late 2023. Weaknesses include overfitting when preferences are near-deterministic, sensitivity to whether data are on-policy, a tendency to lower the likelihood of both responses, and exploitation of length. Variants include IPO, cDPO, ORPO, SimPO, online or iterative DPO and KTOKahneman–Tversky optimization, n.: A way to tune an AI's behavior that needs only a thumbs-up or thumbs-down on single answers, rather than side-by-side comparisons, borrowing ideas from the psychology of how people weigh gains and losses..

    We swapped PPO for DPO, and the whole preference stage now trains in an afternoon on one node.

LDPO=− E[log⁡σ ⁣(βlog⁡πθ(yw∣x)πref(yw∣x)−βlog⁡πθ(yl∣x)πref(yl∣x))]\mathcal{L}_{\mathrm{DPO}} = -\,\mathbb{E}\left[\log \sigma\!\left(\beta \log \frac{\pi_\theta(y_w \mid x)}{\pi_{\mathrm{ref}}(y_w \mid x)} - \beta \log \frac{\pi_\theta(y_l \mid x)}{\pi_{\mathrm{ref}}(y_l \mid x)}\right)\right]

Origin

Introduced by Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher Manning and Chelsea Finn of Stanford in May 2023, under the subtitle 'Your Language Model is Secretly a Reward Model' (NeurIPS 2023). Uptake was rapid in open models such as Zephyr (October 2023) and Mixtral-Instruct, and Meta's Llama 3 (2024) used DPO rather than PPO in post-training.

Usage note

Whether DPO matches well-tuned PPO-based RLHF is debated: several studies found online RL ahead on hard tasks, others found little gap at modest scale. DPO belongs to the family called direct alignment algorithmsdirect alignment algorithm, n.: Any method that tunes an AI directly on examples of preferred and dispreferred answers, skipping the separate scoring model and trial-and-error training used in classic RLHF..

Related

See also
  • reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
  • preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like.
  • reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started.
  • KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer.
  • Kahneman–Tversky optimizationKahneman–Tversky optimization, n.: A way to tune an AI's behavior that needs only a thumbs-up or thumbs-down on single answers, rather than side-by-side comparisons, borrowing ideas from the psychology of how people weigh gains and losses.
  • Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths.
  • direct alignment algorithmdirect alignment algorithm, n.: Any method that tunes an AI directly on examples of preferred and dispreferred answers, skipping the separate scoring model and trial-and-error training used in classic RLHF.
Contrast with
  • proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback.
  • reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples.

Sources

  1. Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C. D. & Finn, C. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. arXiv:2305.18290. arxiv.org/abs/2305.18290v1

Mentioned in

fine-tuningfine-tuning, n.: Taking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way., post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner., reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average., instruction datasetinstruction dataset, n.: A collection of example requests paired with good answers, used to teach a pretrained language model to follow instructions and act as a helpful assistant., instruction-tuned modelinstruction-tuned model, n.: A language model given extra training on examples of instructions paired with good responses, so it does what you ask instead of merely continuing your text., Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right., objective functionobjective function, n.: The score that an optimization process tries to push as low, or as high, as possible. For an AI, it defines what 'doing well' means during training., preference datapreference data, n.: Records of which of two or more AI answers people (or other AIs) liked better, collected so that a model can learn what good answers look like., proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback., reward overoptimizationreward overoptimization, n.: What happens when an AI is trained too hard to please an imperfect scoring model: its scores keep climbing while the real quality of its answers stops improving and then gets worse., abliterationabliteration, n.: A technique for removing a chat model's ability to refuse requests by finding the internal direction responsible for refusals and permanently erasing it from the model's weights, with no retraining., Bradley–Terry modelBradley–Terry model, n.: A simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths., Chelsea FinnChelsea Finn, prop. n.: An American computer scientist at Stanford known for 'learning to learn' methods that let AI adapt to new tasks from a few examples, and a co-founder of the robotics company Physical Intelligence., direct alignment algorithmdirect alignment algorithm, n.: Any method that tunes an AI directly on examples of preferred and dispreferred answers, skipping the separate scoring model and trial-and-error training used in classic RLHF., Kahneman–Tversky optimizationKahneman–Tversky optimization, n.: A way to tune an AI's behavior that needs only a thumbs-up or thumbs-down on single answers, rather than side-by-side comparisons, borrowing ideas from the psychology of how people weigh gains and losses., KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer., reference modelreference model, n.: A frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started., REINFORCE leave-one-outREINFORCE leave-one-out, n.: A simple reinforcement-learning method that has a model answer each question several times and judges each answer against the average score of the other answers., RewardBenchRewardBench, prop. n.: A public test that checks whether the scoring models used to train chatbots can tell a better answer from a worse one, by counting how often they prefer the right response.