Skip to content
Last Dictionary
Plane
Evaluation & BenchmarksAgents & HarnessesAGI & Superintelligence

MLE-bench

proper noun

In plain wordsA test that gives AI agents 75 real data-science competitions from Kaggle and checks whether their solutions would have won medals against human teams.

  1. A benchmark that tests whether AI agentsAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. can do the whole job of a machine-learning engineer — reading a task, preparing the data, training and tuning models and submitting predictions — on 75 past KaggleKaggle, prop. n.: A popular website, owned by Google, where people compete to build the most accurate prediction models on shared datasets, publish their code and data, and learn data science. competitions in tabular prediction, computer vision, natural-language processing and signal processing. Each competition is reproduced offline with its dataset, a grading script and a snapshot of the human leaderboard, so an agent's submission can be ranked against the original competitors and awarded the bronze, silver or gold medal it would have earned. The headline metric is the share of competitions in which an agent wins any medal. When it was released, the strongest configuration tested, OpenAI's o1-preview running in the AIDE agent scaffold, earned a bronze medal or better in 16.9% of the competitions, and did markedly better when allowed more attempts and time. It is used to track progress toward automated AI researchautomated AI research, n.: Using AI systems to do the work of AI researchers, such as running experiments and writing code for new models, so that AI helps build its own successors..

    With 24 hours per competition, the agent medals on MLE-bench about twice as often as with eight.

Origin

Introduced by Jun Shern Chan, Neil Chowdhury, Oliver Jaffe and colleagues at OpenAI in October 2024, and presented at ICLR 2025. OpenAI uses it in its Preparedness FrameworkPreparedness Framework, prop. n.: OpenAI's published process for testing its AI models for catastrophic risks — such as help with biological weapons or cyberattacks — and deciding what safeguards are needed before release. evaluations of AI self-improvement.

Usage note

Kaggle competitions and winning write-ups are public, so contamination is a known risk; the authors supply tools to detect plagiarism and prior familiarity with a competition.

Related

See also
  • KaggleKaggle, prop. n.: A popular website, owned by Google, where people compete to build the most accurate prediction models on shared datasets, publish their code and data, and learn data science.
  • automated AI researchautomated AI research, n.: Using AI systems to do the work of AI researchers, such as running experiments and writing code for new models, so that AI helps build its own successors.
  • RE-BenchRE-Bench, prop. n.: A test from the research group METR that pits AI agents against human experts on seven open-ended machine-learning engineering projects, comparing what each achieves in the same time.
  • AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
  • Preparedness FrameworkPreparedness Framework, prop. n.: OpenAI's published process for testing its AI models for catastrophic risks — such as help with biological weapons or cyberattacks — and deciding what safeguards are needed before release.
  • dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself.

Sources

  1. Chan, J. S., Chowdhury, N., Jaffe, O. et al. (2024). MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering. arXiv:2410.07095. arxiv.org/abs/2410.07095

Mentioned in

automated AI researchautomated AI research, n.: Using AI systems to do the work of AI researchers, such as running experiments and writing code for new models, so that AI helps build its own successors., RE-BenchRE-Bench, prop. n.: A test from the research group METR that pits AI agents against human experts on seven open-ended machine-learning engineering projects, comparing what each achieves in the same time.