Skip to content
Last Dictionary
Plane
Language Models & TransformersInference & EfficiencyEvaluation & Benchmarks

self-consistency

nounAlso: self-consistency decoding, majority voting, maj@k

In plain wordsAsking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts.

  1. 1

    Decoding

    A decodingdecoding, n.: The method a language model uses to turn its predictions into actual text — for example, always picking the most likely next word, or choosing somewhat at random among the likely ones. method for chain-of-thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate. reasoning that replaces a single greedily decoded solution with a vote: the model works the same problem many times at nonzero temperaturetemperature, n.: A setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone., the final answer is pulled out of each attempt, and the answer reached most often wins (majority vote, reported as maj@k). It works because sound reasoning tends to arrive at the same answer by different routes, whereas mistakes usually lead to scattered, disagreeing answers. It requires answers that can be compared for equality, such as numbers or multiple-choice letters; variants for open-ended text ask a model to select the most consistent response. Cost grows linearly with the number of samples, making it a basic form of test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter. scaling and a standard baseline for reasoning modelsreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning..

    Self-consistency over 32 samples beat the single greedy answer by nine points on GSM8K.

  2. 2

    Evaluation

    More generally, the property that a model gives compatible answers to equivalent or logically related questions, measured in consistency and robustness evaluations; disagreement across paraphrases of the same question signals unreliable knowledge.

Origin

Introduced by Xuezhi Wang, Jason Wei and colleagues at Google in 'Self-Consistency Improves Chain of Thought Reasoning in Language Models' (March 2022; ICLR 2023), which reported gains such as 17.9 percentage points on GSM8K over standard chain-of-thought prompting.

Related

See also
  • chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate.
  • test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter.
  • best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers.
  • samplingsampling, n.: Picking some items at random from a larger set or from a range of possibilities — surveying 1,000 voters, say, or letting an AI choose its next word by a weighted lottery.
  • tree of thoughtstree of thoughts, n.: A method in which an AI explores several possible next steps of a problem, judges which look promising and backs out of dead ends, like a chess player weighing moves, instead of reasoning in one straight line.
  • semantic entropysemantic entropy, n.: A way to tell when an AI is likely making something up: ask it the same question several times and check whether the answers mean the same thing. Answers that keep changing signal guessing.
Contrast with
  • greedy decodinggreedy decoding, n.: The simplest way for a language model to write: always pick the single most likely next word. It is fast and predictable but can produce dull or repetitive text.

Sources

  1. Wang, X., Wei, J., Schuurmans, D., Le, Q., Chi, E., Narang, S., Chowdhery, A. & Zhou, D. (2023). Self-Consistency Improves Chain of Thought Reasoning in Language Models. ICLR 2023. arxiv.org/abs/2203.11171v4

Mentioned in

chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate., AIMEAIME, prop. n.: A hard US high-school math contest whose 15 questions each have a whole-number answer; AI labs use each year's new papers to test their models' mathematical reasoning., best-of-n samplingbest-of-n sampling, n.: Asking an AI the same question several times and keeping whichever answer a scoring model or checker rates highest — a simple way to trade extra computing for better answers., GSM8KGSM8K, prop. n.: A set of about 8,500 grade-school math word problems, each needing a few steps of simple arithmetic, used to test whether language models can reason through a problem step by step., pass@kpass@k, n.: A score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code., samplingsampling, n.: Picking some items at random from a larger set or from a range of possibilities — surveying 1,000 voters, say, or letting an AI choose its next word by a weighted lottery., test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter., model cascademodel cascade, n.: A chain of AI models ordered from cheapest to most capable: each request goes to the cheap one first and moves up the chain only if the answer does not look good enough., semantic entropysemantic entropy, n.: A way to tell when an AI is likely making something up: ask it the same question several times and check whether the answers mean the same thing. Answers that keep changing signal guessing., tree of thoughtstree of thoughts, n.: A method in which an AI explores several possible next steps of a problem, judges which look promising and backs out of dead ends, like a chess player weighing moves, instead of reasoning in one straight line.