Humanity's Last Exam
In plain wordsA test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%.
A benchmark of 2,500 questions written by subject experts worldwide, built to outlast earlier tests that frontier models had saturated: each question has one unambiguous, checkable answer, and the set spans dozens of academic fields — about 41% mathematics, with the rest spread over biology and medicine, computer science, physics, the humanities, chemistry, engineering and other subjects. About 14% of questions include images; about 24% are multiple choice and the rest short-answer, graded by exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters. with a model-based judge. Candidate questions were kept only if leading models failed them before expert review, so initial scores were very low, and the benchmark also reports calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time. error because models tended to give wrong answers with high confidence. A private held-out set guards against overfitting. Top scores rose from under 10% at release to above 35% by the end of 2025 and, as of October 2026, to about 55% on the official leaderboard kept by Scale AI. A 2025 audit by FutureHouse suggested that roughly 30% of chemistry and biology answers might be wrong; the organizers acknowledged some errors and launched a continuously revised fork, HLE-Rolling, followed in September 2026 by HLE-Diamond.
The new model scores 45% on Humanity's Last Exam with tools, but its calibration error is still above 40%.
Origin
Created by the Center for AI Safety and Scale AI, led by Dan Hendrycks with Long Phan, Alice Gatti and colleagues, from questions by nearly 1,000 contributors at more than 500 institutions in 50 countries. Released in January 2025 and published in Nature in January 2026. Hendrycks has said a conversation with Elon Musk, who found existing tests too easy, prompted the project.
Usage note
The name presents it as the last exam of its kind that would be needed. A high score would show expert-level performance on closed questions with checkable answers — not autonomous research ability or artificial general intelligenceartificial general intelligence, n.: AI that could learn and do almost any mental task a capable adult can, rather than excelling at one narrow job. Experts disagree on the exact bar and on when, or whether, it will arrive., as the organizers themselves say.
Related
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress.
- MMLUMMLU, prop. n.: A giant multiple-choice exam for AI covering 57 subjects, from school math and history to law and medicine; for years it was the most quoted test of a model's general knowledge.
- GPQAGPQA, prop. n.: A set of very hard multiple-choice science questions written by PhD-level experts, designed so that looking things up online does not help much: capable non-specialists with web access still mostly fail.
- FrontierMathFrontierMath, prop. n.: Several hundred new math problems, kept unpublished so AI systems cannot have trained on them, written by professional mathematicians to check whether machines can do advanced, even research-level, mathematics.
- calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
- exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters.
Sources
- Phan, L., Gatti, A., Han, Z., Li, N. et al. (2025). Humanity's Last Exam. arXiv:2501.14249. arxiv.org/abs/2501.14249
- Center for AI Safety, Scale AI & HLE Contributors Consortium (2026). A benchmark of expert-level academic questions to assess AI capabilities. Nature 649, 1139–1146. doi.org/10.1038/s41586-025-09962-4
- Scale AI. Humanity's Last Exam leaderboard. labs.scale.com/leaderboard/humanitys_last_exam
Mentioned in
deep researchdeep research, n.: An AI feature that spends minutes rather than seconds on a question — planning, running dozens of web searches, reading many sources — and then writes a long report with citations., benchmark saturationbenchmark saturation, n.: When the best AI systems score so close to the maximum on a test that it can no longer tell them apart or show further progress., Center for AI SafetyCenter for AI Safety, prop. n.: A San Francisco nonprofit that studies how advanced AI could lead to disaster and how to prevent it. In 2023 it gathered signatures for a one-sentence warning that AI could threaten human extinction., Dan HendrycksDan Hendrycks, prop. n.: An American AI-safety researcher who created widely used AI tests such as MMLU and Humanity's Last Exam, and who runs the Center for AI Safety., exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters., GPQAGPQA, prop. n.: A set of very hard multiple-choice science questions written by PhD-level experts, designed so that looking things up online does not help much: capable non-specialists with web access still mostly fail., MMLUMMLU, prop. n.: A giant multiple-choice exam for AI covering 57 subjects, from school math and history to law and medicine; for years it was the most quoted test of a model's general knowledge., Scale AIScale AI, prop. n.: A company that pays large numbers of people to label examples, write model answers and grade chatbot replies, producing the data AI developers train and test on. Meta bought nearly half of it in 2025., Alexandr WangAlexandr Wang, prop. n.: An American entrepreneur who founded Scale AI, a major supplier of human-labeled data for training AI, and became Meta's chief AI officer in 2025., canary stringcanary string, n.: A unique marker planted in data to trace what an AI model has absorbed: a code in test files so they can be kept out of training, or a fake secret hidden in training data to measure memorization., Virology Capabilities TestVirology Capabilities Test, prop. n.: A hard test of practical laboratory virology know-how, used to check whether AI models know enough about working with viruses that they might help someone misuse biology.