FrontierMath
In plain wordsSeveral hundred new math problems, kept unpublished so AI systems cannot have trained on them, written by professional mathematicians to check whether machines can do advanced, even research-level, mathematics.
A benchmark of previously unpublished mathematics problems, each composed and checked by professional mathematicians, that samples most major branches of the field, including computation-heavy number theory and combinatorics as well as abstract areas such as category theory and algebraic geometry. Each problem has a single definite answer — typically a number or explicit mathematical object — that a computer can check automatically, so answers are hard to guess and grading needs no human. Problems are grouped into Tiers 1–3 (advanced undergraduate to graduate difficulty) and Tier 4 (research level). Most are kept private and run by Epoch AI, making the benchmark contamination-resistantcontamination-resistant benchmark, n.: A test built so that its questions are unlikely to have been seen by an AI during training — kept secret, refreshed often, or written after the model was made.. A June 2026 revision corrected errors found in about 42% of problems and removed a dozen, leaving 338 (295 in Tiers 1–3 and 43 in Tier 4). At release in November 2024, leading models solved under 2%. Epoch has since added companion sets, including FrontierMath: Open Problems, unsolved research questions whose proposed solutions a program can verify.
FrontierMath scores tripled in a year, though most of the gain came on Tier 1 problems.
Origin
Developed by Epoch AI with dozens of mathematicians; the paper, led by Elliot Glazer, appeared in November 2024. Epoch later disclosed that OpenAI had funded the benchmark and had access to much of the problem set apart from a holdout, drawing criticism of the initial lack of transparency.
Usage note
Answer-checking tests whether a model reaches the right result, not whether it can write a rigorous proof. Because one developer funded it and holds part of it, Epoch's independent runs on private sets are more reliable than self-reported scores.
Related
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- contamination-resistant benchmarkcontamination-resistant benchmark, n.: A test built so that its questions are unlikely to have been seen by an AI during training — kept secret, refreshed often, or written after the model was made.
- MATHMATH, prop. n.: A set of 12,500 problems from high-school math competitions, each with a full worked solution, used to test language models on genuinely challenging mathematics.
- AIMEAIME, prop. n.: A hard US high-school math contest whose 15 questions each have a whole-number answer; AI labs use each year's new papers to test their models' mathematical reasoning.
- IMO Grand ChallengeIMO Grand Challenge, prop. n.: The goal of building an AI that earns a gold medal at the International Mathematical Olympiad, the top contest for school-age mathematicians. AI systems first reached gold-medal scores in 2025, though not under the challenge's strict original rules.
- automated theorem provingautomated theorem proving, n.: Getting computers to prove mathematical or logical statements by themselves, by searching for a chain of valid steps that leads from basic assumptions to the statement.
- reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning.
Sources
- Glazer, E. et al. (2024). FrontierMath: A Benchmark for Evaluating Advanced Mathematical Reasoning in AI. arXiv:2411.04872. arxiv.org/abs/2411.04872
- Epoch AI. FrontierMath benchmark hub. epoch.ai/benchmarks/frontiermath
Mentioned in
Humanity's Last ExamHumanity's Last Exam, prop. n.: A test of 2,500 extremely hard questions with definite answers, written by about a thousand experts worldwide and billed as the last academic exam AI would need; at launch, top models scored under 10%., AIMEAIME, prop. n.: A hard US high-school math contest whose 15 questions each have a whole-number answer; AI labs use each year's new papers to test their models' mathematical reasoning., Epoch AIEpoch AI, prop. n.: A research nonprofit that gathers data on how fast AI is advancing — the computing power, data and money behind it — and publishes forecasts and difficult tests for AI., International Mathematical OlympiadInternational Mathematical Olympiad, prop. n.: The world's top math competition for high-school students. Because its problems demand creative written proofs, not just answers, earning a gold medal there became a famous test of AI reasoning, first passed in 2025., MATHMATH, prop. n.: A set of 12,500 problems from high-school math competitions, each with a full worked solution, used to test language models on genuinely challenging mathematics., contamination-resistant benchmarkcontamination-resistant benchmark, n.: A test built so that its questions are unlikely to have been seen by an AI during training — kept secret, refreshed often, or written after the model was made., IMO Grand ChallengeIMO Grand Challenge, prop. n.: The goal of building an AI that earns a gold medal at the International Mathematical Olympiad, the top contest for school-age mathematicians. AI systems first reached gold-medal scores in 2025, though not under the challenge's strict original rules.