METR
In plain wordsA nonprofit research group that checks new AI systems for dangerous skills, best known for tracking how long the tasks are that AI agents can now finish on their own.
A Berkeley, California, nonprofit that designs and runs tests of frontier AI systems for the abilities its researchers consider most relevant to catastrophe: acting on their own over long stretches and doing the work of AI researchers. It has carried out pre-deployment testingpre-deployment testing, n.: Checking a new AI model for dangerous abilities and bad behavior before it is released to the public, sometimes with outside experts or government testers given early access. of models from OpenAI and Anthropic, contributing to their system cardssystem card, n.: A public report released with an AI model that explains what it can do, how it was tested for safety, what risks were found and what protections were put in place., and builds task suites such as RE-BenchRE-Bench, prop. n.: A test from the research group METR that pits AI agents against human experts on seven open-ended machine-learning engineering projects, comparing what each achieves in the same time. for research-engineering ability. Its best-known measure, the task-completion time horizonMETR time horizon, n.: A way of measuring AI progress by asking how long a task — timed by how long it takes a skilled human — an AI agent can complete with even odds of success. (March 2025), estimates the length of task, in skilled-human working time, that an AI agent completes with 50 percent reliability; that length had doubled roughly every seven months since 2019, and METR's later measurements pointed to faster doubling, about every four months, from 2024. Its widely cited 2025 randomized trial found experienced open-source developers about 19 percent slower when allowed early-2025 AI tools. Its staff also helped formulate the idea of the responsible scaling policyresponsible scaling policy, n.: A public rulebook in which an AI company promises to test each new model for dangerous abilities and to add stronger safeguards, or slow down, as its models become more capable. (2023).
The new model's METR time horizon came in at several hours.
Origin
Began in 2022 as ARC Evals, the evaluation team that Beth Barnes, formerly of OpenAI, set up inside Paul Christiano's Alignment Research CenterAlignment Research Center, prop. n.: A small Berkeley nonprofit started by Paul Christiano that searches for mathematical ways to keep future AI systems, even ones far smarter than people, honest and under human control.; its tests of GPT-4 before release drew wide attention in 2023. In December 2023 it became an independent nonprofit in Berkeley, California, renamed METR, with Barnes as chief executive.
Related
- METR time horizonMETR time horizon, n.: A way of measuring AI progress by asking how long a task — timed by how long it takes a skilled human — an AI agent can complete with even odds of success.
- dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself.
- pre-deployment testingpre-deployment testing, n.: Checking a new AI model for dangerous abilities and bad behavior before it is released to the public, sometimes with outside experts or government testers given early access.
- RE-BenchRE-Bench, prop. n.: A test from the research group METR that pits AI agents against human experts on seven open-ended machine-learning engineering projects, comparing what each achieves in the same time.
- Alignment Research CenterAlignment Research Center, prop. n.: A small Berkeley nonprofit started by Paul Christiano that searches for mathematical ways to keep future AI systems, even ones far smarter than people, honest and under human control.
- third-party AI evaluationthird-party AI evaluation, n.: Testing of an AI system by someone other than the company that made it — a government institute, a nonprofit lab or independent researchers — to check what it can do and how risky it is.
Sources
- Kwa, T. et al. (2025). Measuring AI Ability to Complete Long Tasks. METR / arXiv.
Mentioned in
GitHub CopilotGitHub Copilot, prop. n.: An AI assistant from GitHub, built into programmers' code editors, that suggests code as they type, answers questions about code and can carry out coding tasks on its own., dangerous capability evaluationdangerous capability evaluation, n.: A test of whether an AI system can do things that could cause severe harm in the wrong hands or on its own — such as helping make a bioweapon, hacking computers or copying itself., Epoch AIEpoch AI, prop. n.: A research nonprofit that gathers data on how fast AI is advancing — the computing power, data and money behind it — and publishes forecasts and difficult tests for AI., GPT-4GPT-4, prop. n.: OpenAI's 2023 language model, a big step up from the one behind the first ChatGPT. It scored well on many professional and academic exams and could also read images., Paul ChristianoPaul Christiano, prop. n.: An American AI-alignment researcher who led the work that made training AI from human preferences practical, the basis of how chatbots are taught to be helpful., third-party AI evaluationthird-party AI evaluation, n.: Testing of an AI system by someone other than the company that made it — a government institute, a nonprofit lab or independent researchers — to check what it can do and how risky it is., Alignment Research CenterAlignment Research Center, prop. n.: A small Berkeley nonprofit started by Paul Christiano that searches for mathematical ways to keep future AI systems, even ones far smarter than people, honest and under human control., Apollo ResearchApollo Research, prop. n.: An AI safety organization, founded in London in 2023, that tests advanced AI systems for signs that they might secretly pursue goals their developers did not intend., autonomous replication and adaptationautonomous replication and adaptation, n.: An AI's ability to keep itself running and spreading without human help — earning money, renting computers, copying itself and fixing problems — which safety testers check for., capability elicitationcapability elicitation, n.: The work of drawing out the best an AI can do on a test — with better instructions, tools, retries or extra training — so that its abilities are not underestimated., self-exfiltrationself-exfiltration, n.: When an AI model copies its own underlying files — its learned weights — out of the lab that controls it, so that it can keep running beyond its developers' oversight.