superficial alignment hypothesis
In plain wordsThe idea that a language model learns nearly all its knowledge and skills from its initial bulk training, and that later tuning mainly teaches it the style and format to use when talking to people.
The conjecture, in the wording of the 2023 LIMA paper, that a model's 'knowledge and capabilities are learnt almost entirely during pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.', so that alignment-oriented post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner. of a large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. mainly chooses which already-learned style of response — format, register, persona — to use with users. The evidence came from LIMA itself: a 65-billion-parameter LLaMA model given only 1,000 hand-picked training examples and no reinforcement learning, whose answers human raters rated as good as or better than GPT-4's in 43 percent of head-to-head comparisons. Supporting analyses found that aligned and base models differ mostly on stylistic tokens, and that in-context examples alone can approximate alignment. Critics reply that preference evaluations reward style; that post-training measurably adds abilities such as reliable tool use, long-horizon reasoning and calibrated refusal; and that large-scale reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds. since 2024 produces gains hard to call superficial.
If the superficial alignment hypothesis holds, a thousand excellent examples should matter more than a million mediocre ones.
Origin
Proposed by Chunting Zhou and colleagues, led from Meta AI, in the LIMA paper (May 2023; NeurIPS 2023), which stated that alignment teaches a model which subdistribution of formats to use with users. Bill Yuchen Lin and colleagues (December 2023) offered supporting evidence from token distributions.
Usage note
A hypothesis about where capabilities come from, not a claim that alignment is easy or unimportant; 'superficial' refers to how deeply post-training changes the model, not to its practical value.
Related
- post-trainingpost-training, n.: Everything done to an AI model after its initial bulk training on huge amounts of text: the extra teaching that turns a raw text predictor into a helpful, safe assistant or a careful reasoner.
- supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples.
- instruction tuninginstruction tuning, n.: Training a language model on many tasks written as plain instructions with correct answers, so it learns to do what a request asks instead of just continuing the text.
- pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.
- base modelbase model, n.: A language model fresh from its first broad training on text, before being taught to follow instructions or chat. It continues whatever text it is given instead of answering.
- reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.
- reinforcement learning with verifiable rewardsreinforcement learning with verifiable rewards, n.: Training an AI by letting it attempt problems whose answers can be checked automatically — a math result, a passing program — and rewarding it only when the check succeeds.
Sources
- Zhou, C. et al. (2023). LIMA: Less Is More for Alignment. NeurIPS 2023. arXiv:2305.11206. arxiv.org/abs/2305.11206
Mentioned in
supervised fine-tuningsupervised fine-tuning, n.: Teaching a pretrained AI model by showing it many examples of good answers to requests, so it learns to imitate them — like learning a job by studying worked examples.