benchmark saturation
In plain wordsWhen the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
The state in which leading systems approach a benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.'s ceiling, so that remaining differences fall within noise and the benchmark no longer discriminates between models or tracks progress. The effective ceiling is often below 100%, set by mislabeled or ambiguous items — about 6.5% of MMLUMMLU, prop. n.: A giant multiple-choice exam for AI covering 57 subjects, from school math and history to law and medicine; for years it was the most quoted test of a model's general knowledge. questions are estimated to contain errors — and by genuine disagreement among experts. Saturation has accelerated: GLUEGLUE, prop. n.: A 2018 bundle of nine English language-understanding tests combined into one score, which became the yardstick for the first wave of pretrained language models such as BERT. was overtaken by systems scoring above its human baseline within about a year of release, prompting SuperGLUESuperGLUE, prop. n.: A harder follow-up to the GLUE language-understanding test, released in 2019 after AI systems had already beaten the human baseline on the original.; MMLU, GSM8KGSM8K, prop. n.: A set of about 8,500 grade-school math word problems, each needing a few steps of simple arithmetic, used to test whether language models can reason through a problem step by step. and HumanEvalHumanEval, prop. n.: A set of 164 small programming exercises, each described in plain English and checked by hidden tests, used to see whether an AI can write working Python code. were near their ceilings by 2024; and in February 2026 OpenAI stopped reporting SWE-benchSWE-bench, prop. n.: A test that asks AI to fix real bugs and add features in real open-source Python projects, checked by running each project's own tests; the best-known yardstick for AI coding agents. Verified, citing contamination and flawed tests. Responses include harder successors, pruning of solved items, private test sets and contamination-resistantcontamination-resistant benchmark, n.: A test built so that its questions are unlikely to have been seen by an AI during training — kept secret, refreshed often, or written after the model was made. designs.
GSM8K is saturated — every frontier model is within two points of the noise ceiling — so we dropped it from the report.
Origin
The word was applied to benchmarks by the late 2010s, as GLUE was overtaken within about a year of its 2018 release. Douwe Kiela and colleagues' Dynabench paper (2021) charted how quickly successive benchmarks reached human-level scores, and by 2024 'saturated' was routine in model reports, benchmark papers and policy documents.
Usage note
Distinct from, but entangled with, data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.: a benchmark can be saturated legitimately, or only appear so because its items leaked into training data. A high score on a saturated benchmark need not show the capability its name suggests.
Related
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability.
- contamination-resistant benchmarkcontamination-resistant benchmark, n.: A test built so that its questions are unlikely to have been seen by an AI during training — kept secret, refreshed often, or written after the model was made.
- Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%.
- construct validityconstruct validity, n.: Whether high marks on a test actually reflect the skill the test is named after — for example, whether a 'reasoning' test rewards reasoning rather than memory or test-taking tricks.
- Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure.
- benchmaxxingbenchmaxxing, n.: Slang for tuning an AI model to score as high as possible on well-known tests, even when this does not make it genuinely better at real-world tasks.
Sources
- Kiela, D. et al. (2021). Dynabench: Rethinking Benchmarking in NLP. NAACL 2021.
- OpenAI (2026). Why we no longer evaluate SWE-bench Verified. openai.com/index/why-we-no-longer-evaluate-swe-bench-verified
Mentioned in
benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time., Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%., leaderboardleaderboard, n.: A public ranking table showing which AI systems score best on a test, updated as new results arrive — like a sports league table for models., SWE-benchSWE-bench, prop. n.: A test that asks AI to fix real bugs and add features in real open-source Python projects, checked by running each project's own tests; the best-known yardstick for AI coding agents., construct validityconstruct validity, n.: Whether high marks on a test actually reflect the skill the test is named after — for example, whether a 'reasoning' test rewards reasoning rather than memory or test-taking tricks., GLUEGLUE, prop. n.: A 2018 bundle of nine English language-understanding tests combined into one score, which became the yardstick for the first wave of pretrained language models such as BERT., Goodhart's lawGoodhart's law, n.: The rule of thumb that once a number is used as a target, people or AIs find ways to push it up that no longer reflect what it was meant to measure., GPQAGPQA, prop. n.: A set of very hard multiple-choice science questions written by PhD-level experts, designed so that looking things up online does not help much: capable non-specialists with web access still mostly fail., GSM8KGSM8K, prop. n.: A set of about 8,500 grade-school math word problems, each needing a few steps of simple arithmetic, used to test whether language models can reason through a problem step by step., HumanEvalHumanEval, prop. n.: A set of 164 small programming exercises, each described in plain English and checked by hidden tests, used to see whether an AI can write working Python code., International Mathematical OlympiadInternational Mathematical Olympiad, prop. n.: The world's top math competition for high-school students. Because its problems demand creative written proofs, not just answers, earning a gold medal there became a famous test of AI reasoning, first passed in 2025., MMLUMMLU, prop. n.: A giant multiple-choice exam for AI covering 57 subjects, from school math and history to law and medicine; for years it was the most quoted test of a model's general knowledge., MNISTMNIST, prop. n.: A famous collection of 70,000 small images of handwritten digits used to test image-recognition programs. Getting a model to read MNIST digits is the 'hello world' of machine learning., MTEBMTEB, prop. n.: A large public test suite that scores AI models which turn text into lists of numbers on many different jobs, such as searching, grouping, classifying and matching translations, so people can compare them on one leaderboard., SQuADSQuAD, prop. n.: A Stanford collection of more than 100,000 questions about Wikipedia articles, each solved by highlighting the words in the article that give the answer; it drove rapid progress in machine reading from 2016., state of the artstate of the art, n.: The best performance or most advanced technique currently known in a field. In AI papers, a state-of-the-art result is one that beats every earlier published result on a test., contamination-resistant benchmarkcontamination-resistant benchmark, n.: A test built so that its questions are unlikely to have been seen by an AI during training — kept secret, refreshed often, or written after the model was made., Epoch Capabilities IndexEpoch Capabilities Index, prop. n.: A single number summarizing an AI model's general capability, calculated by the research group Epoch AI by combining its scores on dozens of tests onto one common scale., HellaSwagHellaSwag, prop. n.: A test in which an AI must pick the most sensible ending for a description of an everyday scene; the wrong endings were machine-written to fool computers, so people found it easy and early AI did not., item response theoryitem response theory, n.: A set of statistical models from educational testing that work out both how able each test-taker is and how hard each question is from who answered what correctly; now also used to evaluate AI models., LibriSpeechLibriSpeech, prop. n.: A free collection of about 1,000 hours of volunteers reading English audiobooks, matched with the book text; it became the standard yardstick for speech-recognition systems., miniF2FminiF2F, prop. n.: A test of 488 math problems, mostly from competitions, written in a form a computer can check, used to measure how well AI systems produce proofs that are verified automatically., PutnamBenchPutnamBench, prop. n.: A test built from hundreds of problems from the Putnam, North America's top university math contest, written in a form a computer can check, used to measure how well AI systems prove harder mathematics., RewardBenchRewardBench, prop. n.: A public test that checks whether the scoring models used to train chatbots can tell a better answer from a worse one, by counting how often they prefer the right response.