GSM8K
In plain wordsA set of about 8,500 grade-school math word problems, each needing a few steps of simple arithmetic, used to test whether language models can reason through a problem step by step.
A dataset of about 8,500 grade-school math word problems written by human contributors in deliberately varied wording and split into 7,473 training and 1,319 test items. Each needs two to eight elementary calculations and comes with a worked solution in natural language ending in a numerical answer, which is scored by exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters.. It became the standard test of multi-step arithmetic reasoning: chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate. prompting, self-consistencyself-consistency, n.: Asking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts. and verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded.-based reranking all made their names on it. Scores rose from about 57% for PaLM with chain-of-thought prompting in 2022 to 92% for GPT-4GPT-4, prop. n.: OpenAI's 2023 language model, a big step up from the one behind the first ChatGPT. It scored well on many professional and academic exams and could also read images. in 2023 and above 95% for frontier models by 2024, leaving the benchmark saturatedbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.. Studies using freshly written or templated variants — GSM1k and GSM-Symbolic, both 2024 — found accuracy drops for some models, read as signs of contamination or pattern-matching, while leading models were largely unaffected.
GSM8K is saturated, so we now report math results on AIME and FrontierMath instead.
Origin
Introduced by Karl Cobbe, Vineet Kosaraju and colleagues at OpenAI in 'Training Verifiers to Solve Math Word Problems' (October 2021), which showed that sampling many candidate solutions and choosing the one a trained verifier ranks highest beat fine-tuning alone. The problems were written by human contractors.
Usage note
A small fraction of test answers are wrong or ambiguous, so differences near the ceiling are not meaningful.
Related
- MATHMATH, prop. n.: A set of 12,500 problems from high-school math competitions, each with a full worked solution, used to test language models on genuinely challenging mathematics.
- chain of thoughtchain of thought, n.: Getting an AI to write out its reasoning step by step before giving an answer, the way a student shows their working. This often makes answers to math and logic problems more accurate.
- self-consistencyself-consistency, n.: Asking an AI the same question many times, letting it reason differently each time, then going with the answer it reaches most often — a vote among its own attempts.
- verifierverifier, n.: A program or AI model that checks whether a proposed answer is correct — by running tests, comparing with a known result or judging the reasoning — so that good answers can be picked or rewarded.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
- data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.
Sources
- Cobbe, K., Kosaraju, V., Bavarian, M. et al. (2021). Training Verifiers to Solve Math Word Problems. arXiv:2110.14168. arxiv.org/abs/2110.14168
- Mirzadeh, I. et al. (2025). GSM-Symbolic: Understanding the Limitations of Mathematical Reasoning in Large Language Models. ICLR 2025.
Mentioned in
benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress., data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability., exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters., MATHMATH, prop. n.: A set of 12,500 problems from high-school math competitions, each with a full worked solution, used to test language models on genuinely challenging mathematics., outcome reward modeloutcome reward model, n.: A model that judges only whether an AI's final answer is right, ignoring how it got there — like a teacher who checks just the answer box.