RE-Bench
In plain wordsA test from the research group METR that pits AI agents against human experts on seven open-ended machine-learning engineering projects, comparing what each achieves in the same time.
A benchmark meant to estimate how close frontier models are to automating AI research and development, made of seven open-ended research-engineering environments — for example, optimizing a GPU kernel, speeding up a training script, or fine-tuning a model toward a target behavior — each scored continuously by how far a submission improves on a starting solution. What sets it apart is its human baseline: 61 machine-learning professionals contributed 71 attempts of eight hours each, showing how human results grow with time. Given two hours per environment, the strongest agents at release outscored the experts roughly fourfold; at eight hours the experts edged ahead, and they gained more from extra time — evidence that the agents' strength lay in speed rather than sustained research. RE-Bench tasks form part of the suite behind the METR time horizonMETR time horizon, n.: A way of measuring AI progress by asking how long a task — timed by how long it takes a skilled human — an AI agent can complete with even odds of success. and inform dangerous-capability evaluationsdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself. for automated AI researchautomated AI research, n.: Using AI systems to do the work of AI researchers, such as running experiments and writing code for new models, so that AI helps build its own successors..
On RE-Bench the agent beats humans at two hours, then plateaus while the experts keep improving.
Origin
Introduced by Hjalmar Wijk, Tao Lin, Joel Becker and colleagues at METR (Model Evaluation and Threat Research) in November 2024.
Usage note
With only seven environments, results are noisy, and because scores measure improvement over a baseline, averages across environments depend on how each is normalized.
Related
- METR time horizonMETR time horizon, n.: A way of measuring AI progress by asking how long a task — timed by how long it takes a skilled human — an AI agent can complete with even odds of success.
- MLE-benchMLE-bench, prop. n.: A test that gives AI agents 75 real data-science competitions from Kaggle and checks whether their solutions would have won medals against human teams.
- automated AI researchautomated AI research, n.: Using AI systems to do the work of AI researchers, such as running experiments and writing code for new models, so that AI helps build its own successors.
- dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself.
- AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
- recursive self-improvementrecursive self-improvement, n.: An AI improving its own design or training, so that each improved version is better at making the next improvement. It is the engine usually imagined behind an intelligence explosion.
Sources
- Wijk, H., Lin, T., Becker, J. et al. (2024). RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human experts. arXiv:2411.15114. arxiv.org/abs/2411.15114
Mentioned in
METR time horizonMETR time horizon, n.: A way of measuring AI progress by asking how long a task — timed by how long it takes a skilled human — an AI agent can complete with even odds of success., automated AI researchautomated AI research, n.: Using AI systems to do the work of AI researchers, such as running experiments and writing code for new models, so that AI helps build its own successors., long-horizon tasklong-horizon task, n.: A job that takes many steps and a long time — hours or days of work rather than a single answer — so an AI must stay on track and recover from its own mistakes., METRMETR, prop. n.: A nonprofit research group that checks new AI systems for dangerous skills, best known for tracking how long the tasks are that AI agents can now finish on their own., MLE-benchMLE-bench, prop. n.: A test that gives AI agents 75 real data-science competitions from Kaggle and checks whether their solutions would have won medals against human teams.