InstructGPT

A family of GPT-3GPT-3, prop. n.: A huge 2020 OpenAI language model that could perform new tasks simply from instructions or a few examples written into its prompt, without retraining. It made prompting a new way of programming.-based models fine-tuned by OpenAI to follow user instructions, and the paper describing them, which established the three-stage reinforcement learning from human feedbackreinforcement learning from human feedback, n.: A way of training AI using people's judgments: humans compare pairs of answers, a scoring model learns their preferences, and the AI is then trained to produce answers that score well. recipe for language models: supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples. on labeler-written demonstrations; a reward modelreward model, n.: A separate AI model trained to score responses the way people would, so it can stand in for human judges while another model is trained on millions of examples. trained on rankings of several outputs per prompt; and proximal policy optimizationproximal policy optimization, n.: A widely used method for training AI by trial and error that improves the strategy in small, safe steps; it was the workhorse behind training chatbots like ChatGPT with human feedback. against that reward with a per-token KL penaltyKL penalty, n.: A cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer. (plus, in the PPO-ptx variant, mixed-in pretraining updates to reduce regressions). About 40 contractors wrote demonstrations and comparisons for prompts drawn largely from the OpenAI API. Labelers' rankings showed a striking size effect: answers from the 1.3-billion-parameter InstructGPT beat those of the 175-billion-parameter GPT-3, a model more than 100 times larger. The tuned models also scored as more truthful and produced less toxic text, though they lost some ground on certain academic benchmarks, a cost the authors called an alignment taxalignment tax, n.: The price of caution in AI: whatever performance, money or time a developer sacrifices to make a system safe and aligned, compared with an otherwise identical system built with no such care..

Most open RLHF pipelines still follow the InstructGPT recipe of SFT, reward model, then PPO.