reference model
In plain wordsA frozen copy of an AI model, saved before further training, that the new version is compared against so it does not stray too far from where it started.
In preference optimization and reinforcement-learning fine-tuning, a fixed copy of the starting policy, usually the SFTsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples. model, whose token probabilities serve as an anchor. The KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer. in reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. measures divergence from it, and in direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely. the implicit reward of a response is β times the log-ratio of the policy's probability to the reference model's. Keeping a reference model costs memory and an extra forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction. per sample, which motivated reference-free methods such as ORPO and SimPO; some long reasoning-RL runs periodically reset the anchor to a recent policy checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run.. The choice matters: a weak or mismatched reference distorts implicit rewards and shifts the balance between reward and drift.
We reset the reference model to the latest checkpoint every 500 steps so the policy could keep improving.
Origin
A fixed prior policy appears in KL-regularized RL fine-tuning such as Sequence Tutor (Natasha Jaques and colleagues, 2016) and in OpenAI's RLHF papers (2019–2022), which measured drift from the initial model. 'Reference policy' had earlier uses in RL; the name 'reference model' and the notation π_ref were popularized by the direct preference optimization paper of Rafael Rafailov and colleagues (2023).
Usage note
Not to be confused with the teacher in knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run., which the student imitates rather than merely stays near. In group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average. the reference model enters only through the KL term, and some recipes drop it altogether.
Related
- KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer.
- direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely.
- supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples.
- reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well.
- policypolicy, n.: An AI agent's strategy: the rule it follows to pick an action in each situation. Training in reinforcement learning means improving this rule until it earns high rewards.
- checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run.
- knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.
Sources
- Rafailov, R. et al. (2023). Direct Preference Optimization: Your Language Model is Secretly a Reward Model. NeurIPS 2023. arXiv:2305.18290. arxiv.org/abs/2305.18290v1
Mentioned in
reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well., direct preference optimizationdirect preference optimization, n.: A simpler way to teach an AI people's preferences: instead of building a separate scoring model, it trains directly on pairs of better and worse answers, making the better ones more likely., group relative policy optimizationgroup relative policy optimization, n.: A reinforcement-learning recipe for language models that has the model answer the same question many times and rewards each answer by how much better it did than the group's average., Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right., proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback., supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples., direct alignment algorithmdirect alignment algorithm, n.: Any method that tunes an AI directly on examples of preferred and dispreferred answers, skipping the separate scoring model and trial-and-error training used in classic RLHF., Kahneman–Tversky optimizationKahneman–Tversky optimization, n.: A way to tune an AI's behavior that needs only a thumbs-up or thumbs-down on single answers, rather than side-by-side comparisons, borrowing ideas from the psychology of how people weigh gains and losses., KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer.