Skip to content
Last Dictionary
Plane
Safety & AlignmentTraining & Post-trainingLanguage Models & Transformers

AI debate

nounAlso: debate, AI safety via debate, multi-agent debate

In plain wordsA proposed way to supervise very capable AI: two AI systems argue opposite answers to a question and a human judge decides who argued better, on the theory that lies are easier to expose than to defend.

  1. 1

    Scalable oversight

    A scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking. method in which two copies of a model argue for competing answers in turns and a judge — a human or a weaker model — chooses the winner, with both debaters trained by self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples. to win. The bet is that refuting a lie is easier than defending one, so honest argument wins at equilibrium and the judge need only examine the single most contested step rather than the whole problem. Its proposers drew a parallel with complexity theory: a debate refereed by a polynomial-time judge can in principle decide any problem in PSPACE, while a judge checking answers unaided is limited to NP. Experiments on reading-comprehension questions, where only the debaters see the source text, found debate raised judge accuracy over a single persuasive 'consultant', and more so as debaters grew more persuasive. A known weakness is the obfuscated-arguments problem: a dishonest debater may build an argument whose flaw is real but too hard to locate.

    Judges reading the debate transcripts beat the consultancy baseline by a wide margin.

  2. 2

    Reasoning

    Multi-agent debate: an inference-time technique in which several language-modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. instances propose answers, read one another's reasoning and revise over several rounds before converging. It aims at better accuracy and factuality rather than oversight, and its debaters usually cooperate rather than compete.

Origin

Proposed by Geoffrey Irving, Paul Christiano and Dario Amodei at OpenAI in 'AI safety via debate' (2018), which tested a toy version on MNIST digit classification with a judge shown only pixels the debaters revealed. Experiments with language models followed from 2022 at New York University, Anthropic, Google DeepMind and elsewhere. Yilun Du and colleagues (2023) popularized multi-agent debate for reasoning.

Usage note

Debate assumes judges are moved by valid arguments; human biases and persuasive falsehoods are the central objections. The two senses share a name but not a goal: the safety sense is adversarial and aims at truthful judgment.

Related

See also
  • scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking.
  • iterated amplificationiterated amplification, n.: A proposed training scheme in which a person, helped by many copies of an AI, solves harder problems than either could alone; a new AI then learns to imitate that team, and the cycle repeats.
  • prover–verifier gameprover–verifier game, n.: A training setup in which one AI must justify its answers so clearly that a weaker AI can check them, which also makes the strong AI's work easier for people to check.
  • recursive reward modelingrecursive reward modeling, n.: A plan for supervising advanced AI in stages: AI assistants trained on human feedback help people judge the work of a more capable AI, which then helps judge the next one.
  • sandwichingsandwiching, n.: A way of testing oversight methods today: non-experts, helped by an AI, try to get expert-quality answers out of it, and real experts then check whether they succeeded.
  • self-playself-play, n.: Training an AI by having it play against copies of itself, so it always has an opponent at its own level and can keep improving without human examples.

Sources

  1. Irving, G., Christiano, P. & Amodei, D. (2018). AI safety via debate. arXiv:1805.00899.
  2. Khan, A. et al. (2024). Debating with More Persuasive LLMs Leads to More Truthful Answers. ICML 2024.

Mentioned in

game theorygame theory, n.: The mathematics of strategic decisions, where each player's best move depends on what the others do. It is used to study competition and cooperation among people, firms, nations and AI agents., outer alignmentouter alignment, n.: Getting the goal you train an AI on right in the first place, so that a system scoring perfectly on it would actually be doing what you want., Paul ChristianoPaul Christiano, prop. n.: An American AI-alignment researcher who led the work that made training AI from human preferences practical, the basis of how chatbots are taught to be helpful., scalable oversightscalable oversight, n.: The problem of how people can check and correct AI systems whose work is too difficult, fast or plentiful for humans to judge reliably alone — usually by enlisting AI to help with the checking., factored cognitionfactored cognition, n.: The idea that a hard thinking task can be broken into many small, separately checkable pieces, each simple enough for a person or an AI to handle in a short time., HCHHCH, abbr.: A thought experiment: a person who can hand parts of a problem to copies of themselves, who can hand parts on again, endlessly — the ideal some AI training schemes try to approximate., iterated amplificationiterated amplification, n.: A proposed training scheme in which a person, helped by many copies of an AI, solves harder problems than either could alone; a new AI then learns to imitate that team, and the cycle repeats., prosaic alignmentprosaic alignment, n.: Working on AI safety by assuming that powerful future AI will be built from today's techniques — scaled-up neural networks — rather than from some new breakthrough., prover–verifier gameprover–verifier game, n.: A training setup in which one AI must justify its answers so clearly that a weaker AI can check them, which also makes the strong AI's work easier for people to check., recursive reward modelingrecursive reward modeling, n.: A plan for supervising advanced AI in stages: AI assistants trained on human feedback help people judge the work of a more capable AI, which then helps judge the next one., sandwichingsandwiching, n.: A way of testing oversight methods today: non-experts, helped by an AI, try to get expert-quality answers out of it, and real experts then check whether they succeeded.