weak-to-strong generalization
In plain wordsThe finding that a strong AI trained only on answers from a weaker, error-prone teacher can end up outperforming that teacher — studied as a stand-in for humans supervising smarter-than-human AI.
The tendency of a capable model to surpass its teacher when its only training signal is the imperfect labels of a weaker model, and the experimental paradigm built to measure that effect. It is offered as an analogy for superalignmentsuperalignment, n.: The problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it. — humans (weak) overseeing superhuman models (strong). A pretrainedpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses. GPT-4-class model is, for example, fine-tuned on labels from a GPT-2-class model, and the result is scored by performance gap recovered (PGR): the fraction of the gap between the weak supervisor and a strong 'ceiling' model trained on ground truth that the student closes. Naive fine-tuning recovered part of the gap on language classification tasks, less on chess puzzles and little on reward modeling; adding a loss term that let the student trust its own confident predictions over the supervisor's labels brought the GPT-4-class student close to GPT-3.5 level on the language tasks, and bootstrapping through intermediate-sized models helped on chess. Critics note that a weak model's errors are an imperfect proxy for human errors, and that students can learn to imitate their supervisor's mistakes.
With the confidence loss, PGR on the classification tasks rose sharply, but reward modeling barely moved.
Origin
Introduced by Collin Burns, Pavel Izmailov, Jan Hendrik Kirchner, Jan Leike, Ilya Sutskever and colleagues on OpenAI's Superalignment team in December 2023, alongside a grants program for follow-up research.
Usage note
Related to, but distinct from, weak supervisionweak supervision, n.: Training AI with labels that are cheap but imperfect, such as rough rules of thumb, partial tags or noisy guesses, instead of carefully hand-checked answers. in classical machine learning, where cheap noisy labeling heuristics stand in for hand labels; here the point is the capability gap between supervisor and student. The reverse direction is knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run..
Related
- scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
- superalignmentsuperalignment, n.: The problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it.
- eliciting latent knowledgeeliciting latent knowledge, n.: The unsolved problem of getting an AI to tell us what it actually knows — for instance, that a security camera has been tampered with — rather than what it predicts we would believe.
- sandwichingsandwiching, n.: A way of testing oversight methods today: non-experts, helped by an AI, try to get expert-quality answers out of it, and real experts then check whether they succeeded.
- weak supervisionweak supervision, n.: Training AI with labels that are cheap but imperfect, such as rough rules of thumb, partial tags or noisy guesses, instead of carefully hand-checked answers.
- knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.
- knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.
Sources
- Burns, C., Izmailov, P., Kirchner, J. H. et al. (2023). Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervision. arXiv:2312.09390. arxiv.org/abs/2312.09390
Mentioned in
AI alignmentAI alignment, n.: Making an AI system genuinely try to do what the people it serves want — not a literal-minded, distorted or self-serving version of it. Also the research field working on this., scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking., superalignmentsuperalignment, n.: The problem of keeping AI systems far smarter than humans aligned with human intentions — and the name of OpenAI's short-lived 2023–2024 research team devoted to it., Jan LeikeJan Leike, prop. n.: A German AI-alignment researcher who helped develop training from human feedback, co-led OpenAI's superalignment team, and now leads alignment research at Anthropic., prover–verifier gameprover–verifier game, n.: A training setup in which one AI must justify its answers so clearly that a weaker AI can check them, which also makes the strong AI's work easier for people to check., sandwichingsandwiching, n.: A way of testing oversight methods today: non-experts, helped by an AI, try to get expert-quality answers out of it, and real experts then check whether they succeeded.