expert iteration
In plain wordsA learning loop in which a slow, careful search finds better moves or solutions than the AI's quick instincts, and the AI is then trained to imitate them, so each round's search starts smarter.
A reinforcement-learning scheme that alternates two steps: expert improvement, in which a planner or search procedure — classically Monte Carlo tree searchMonte Carlo tree search, n.: A way for a game-playing program to choose a move by simulating many possible continuations, focusing on the most promising lines, and picking the move that worked out best on average. guided by the current network — makes decisions better than the network alone; and imitation, in which the network (the apprentice) is trained to reproduce the expert's decisions, after which the stronger network guides a stronger search. The scheme gives slow deliberation and fast pattern-matching separate jobs and lets each improve the other, turning computation spent on search into better instant judgments, as in AlphaZeroAlphaZero, prop. n.: A 2017 DeepMind program that taught itself chess, shogi and Go from nothing but the rules by playing against itself, and within hours of training beat the strongest existing programs.. In language-model work, 'expert iteration' usually names a simpler loop: sample many solutions, filter them with a verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded. or reward, and fine-tune on the successes — effectively iterated rejection samplingrejection sampling, n.: Generating many candidate answers and keeping only those that pass a check — a scoring model, a test or a correct final answer — often so the AI can then be trained on its own best work. — used for formal theorem proving and math reasoning and closely related to STaRself-taught reasoner, n.: A training method in which a language model writes step-by-step explanations for problems, keeps the ones that led to correct answers, and learns from them, so it gradually teaches itself to reason. and ReST.
Two rounds of expert iteration on Lean problems doubled the number of theorems the prover could close.
Origin
Named by Thomas Anthony, Zheng Tian and David Barber of University College London in a May 2017 paper (NeurIPS 2017), whose ExIt agent learned the board game Hex from scratch and beat the champion program MoHex 1.0. DeepMind's AlphaGo Zero (2017) used the same principle. Stanislas Polu and colleagues at OpenAI (2022) applied expert iteration to formal mathematics in Lean.
Usage note
The original paper framed the two steps with the dual-process picture of fast and slow thinking (System 1 and System 2System 1 and System 2, phr.: A popular way of describing two modes of thinking: fast, automatic and intuitive (System 1) versus slow, effortful and deliberate (System 2). AI researchers borrow it to contrast instant answers with step-by-step reasoning.). In LLM papers the 'expert' is often just the model plus a filter, with no explicit search.
Related
- Monte Carlo tree searchMonte Carlo tree search, n.: A way for a game-playing program to choose a move by simulating many possible continuations, focusing on the most promising lines, and picking the move that worked out best on average.
- AlphaZeroAlphaZero, prop. n.: A 2017 DeepMind program that taught itself chess, shogi and Go from nothing but the rules by playing against itself, and within hours of training beat the strongest existing programs.
- self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples.
- rejection samplingrejection sampling, n.: Generating many candidate answers and keeping only those that pass a check — a scoring model, a test or a correct final answer — often so the AI can then be trained on its own best work.
- self-taught reasonerself-taught reasoner, n.: A training method in which a language model writes step-by-step explanations for problems, keeps the ones that led to correct answers, and learns from them, so it gradually teaches itself to reason.
- imitation learningimitation learning, n.: Teaching an AI by showing it examples of an expert doing a task, so it learns to copy the behavior instead of discovering it through trial and error.
- self-improvementself-improvement, n.: When an AI gets better by learning from material it produces itself — its own answers, critiques or practice problems — rather than only from new human-made examples.
Sources
- Anthony, T., Tian, Z. & Barber, D. (2017). Thinking Fast and Slow with Deep Learning and Tree Search. NeurIPS 2017. arXiv:1705.08439. arxiv.org/abs/1705.08439
- Polu, S., Han, J. M., Zheng, K., Baksys, M., Babuschkin, I. & Sutskever, I. (2022). Formal Mathematics Statement Curriculum Learning. arXiv preprint.
Mentioned in
AlphaGo ZeroAlphaGo Zero, prop. n.: A program from DeepMind that taught itself to play the board game Go better than any human in 2017, starting from nothing but the rules and learning only by playing millions of games against itself., rejection samplingrejection sampling, n.: Generating many candidate answers and keeping only those that pass a check — a scoring model, a test or a correct final answer — often so the AI can then be trained on its own best work., self-improvementself-improvement, n.: When an AI gets better by learning from material it produces itself — its own answers, critiques or practice problems — rather than only from new human-made examples., iterated amplificationiterated amplification, n.: A proposed training scheme in which a person, helped by many copies of an AI, solves harder problems than either could alone; a new AI then learns to imitate that team, and the cycle repeats., self-taught reasonerself-taught reasoner, n.: A training method in which a language model writes step-by-step explanations for problems, keeps the ones that led to correct answers, and learns from them, so it gradually teaches itself to reason.