ARC-AGI
In plain wordsPuzzles made of colored grids — and, since 2026, small video-game-like worlds — that people solve easily but AI long found hard, designed to test whether a machine can learn a new skill from just a few examples.
A family of benchmarks aimed at fluid intelligence rather than accumulated knowledge — in François Chollet's terms, skill-acquisition efficiency: how quickly a system masters tasks it has never seen. In ARC-AGI-1 and ARC-AGI-2, each task shows a few input–output pairs of colored grids; the solver must infer the transformation rule and produce the output for a new input, usually with two attempts allowed. Tasks assume only 'core knowledge' priors such as objectness, counting and elementary geometry. ARC-AGI-1 has 800 public tasks plus semi-private and private evaluation sets; ARC-AGI-2 (March 2025) has 1,000 training tasks and three 120-task evaluation sets, each task solved by at least two people in a public study, and pairs every score with a cost per task. ARC-AGI-3 (March 2026) is interactive: agents must explore hundreds of hand-made game environments with no instructions and are scored on how efficiently they complete levels compared with humans. The ARC Prize Foundation awards prizes for open-source solutions that run within compute limits.
The model scored 75% on ARC-AGI-1 with heavy test-time compute, but under 5% on ARC-AGI-2 when it launched.
Origin
Introduced by François Chollet, then at Google, as the Abstraction and Reasoning Corpus in his November 2019 paper 'On the Measure of Intelligence'. Kaggle contests followed from 2020; Chollet and Mike Knoop launched the ARC Prize in 2024. In December 2024 OpenAI's o3 scored 75.7% on ARC-AGI-1's semi-private set, and 87.5% using 172 times more compute.
Usage note
Chollet holds that passing ARC-AGI would not by itself demonstrate artificial general intelligenceartificial general intelligence, n.: AI that could learn and do almost any mental task a capable adult can, rather than excelling at one narrow job. Experts disagree on the exact bar and on when, or whether, it will arrive.; critics question how well grid puzzles capture general intelligence. At ARC-AGI-3's launch, frontier systems scored about 0.5% against 100% for humans.
Related
- artificial general intelligenceartificial general intelligence, n.: AI that could learn and do almost any mental task a capable adult can, rather than excelling at one narrow job. Experts disagree on the exact bar and on when, or whether, it will arrive.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter.
- reasoning modelreasoning model, n.: An AI model trained to work through a problem step by step in hidden 'thinking' text before it answers. Letting it think longer generally makes it better at math, coding and planning.
- program synthesisprogram synthesis, n.: Getting a computer to write a program automatically from a description of what the program should do — a logical specification, a few input–output examples or a request in plain language.
- contamination-resistant benchmarkcontamination-resistant benchmark, n.: A test built so that its questions are unlikely to have been seen by an AI during training — kept secret, refreshed often, or written after the model was made.
- KaggleKaggle, prop. n.: A popular website, owned by Google, where people compete to build the most accurate prediction models on shared datasets, publish their code and data, and learn data science.
Sources
- Chollet, F. (2019). On the Measure of Intelligence. arXiv:1911.01547. arxiv.org/abs/1911.01547
- ARC Prize Foundation (2025). Announcing ARC-AGI-2 and ARC Prize 2025. arcprize.org/blog/announcing-arc-agi-2-and-arc-prize-2025
- ARC Prize Foundation (2026). ARC-AGI-3 launch. arcprize.org/blog/arc-agi-3-launch
Mentioned in
artificial general intelligenceartificial general intelligence, n.: AI that could learn and do almost any mental task a capable adult can, rather than excelling at one narrow job. Experts disagree on the exact bar and on when, or whether, it will arrive., intelligenceintelligence, n.: The ability to learn, understand, reason and solve problems — especially new ones — in order to reach goals. Experts disagree about how to define it precisely, for people and even more for machines., Alignment Research CenterAlignment Research Center, prop. n.: A small Berkeley nonprofit started by Paul Christiano that searches for mathematical ways to keep future AI systems, even ones far smarter than people, honest and under human control., contamination-resistant benchmarkcontamination-resistant benchmark, n.: A test built so that its questions are unlikely to have been seen by an AI during training — kept secret, refreshed often, or written after the model was made., François CholletFrançois Chollet, prop. n.: A French software engineer who created Keras, a popular tool for building neural networks, and the ARC benchmark, which tests whether AI can solve kinds of puzzles it has never seen., hierarchical reasoning modelhierarchical reasoning model, n.: A small, brain-inspired AI model that solves puzzles with two linked parts — a slow planner and a fast worker — instead of writing out its reasoning in words., OpenAI o3OpenAI o3, prop. n.: OpenAI's second-generation reasoning model, announced in December 2024. Its high scores on a puzzle test designed to resist AI made headlines, and the released version could use tools such as web search while thinking., test-time trainingtest-time training, n.: Letting an AI model briefly update itself on each new problem it is given, learning from that problem's own clues before answering, instead of keeping its knowledge frozen after training., tiny recursive modeltiny recursive model, n.: A very small AI model that solves hard puzzles by going over its own answer again and again, improving it each time, instead of relying on sheer size.