τ-bench
In plain wordsA test of AI customer-service agents in which a simulated customer, played by another AI, chats with the agent, which must use company tools and follow written rules to resolve the request.
A benchmark that tests whether an AI agent can serve a customer while keeping to company rules, first in retail and airline settings. A large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. plays the customer, holding a hidden goal and disclosing details only gradually, while the agent works through domain-specific API tools (to look up orders, change reservations or issue refunds) and a written policy it must obey. A task counts as solved only if the database ends up matching the annotators' goal state, which forces the agent both to satisfy the user and to stay within policy. The benchmark also introduced pass^k, the chance that all k independent attempts at a task succeed, as a measure of reliability; at release, agents built on GPT-4o solved fewer than half the tasks, and their pass^8 in the retail domain fell below 25%. A successor, τ²-bench (2025), added a telecom domain in which the simulated user also acts on a shared environment.
Our airline agent hits 70% pass^1 on τ-bench, but pass^4 drops below 40%.
Origin
Introduced in June 2024 by Shunyu Yao, Noah Shinn, Pedram Razavi and Karthik Narasimhan, working at Sierra, a company building customer-service agents. Sierra researchers released τ²-bench in 2025.
Usage note
The simulated user is itself a language model, so some failures stem from the simulator rather than the agent, and results vary with the user model chosen.
Related
- AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
- tool usetool use, n.: An AI model's ability to reach beyond its own knowledge by operating outside tools — searching the web, running code, using a calculator or an app — and folding the results into its work.
- function callingfunction calling, n.: A feature of AI model services that lets the model reply with a neatly formatted request to run one of a program's functions, rather than with ordinary text; the program, not the model, then runs it.
- pass@kpass@k, n.: A score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code.
- agentic commerceagentic commerce, n.: Shopping and payments carried out by AI agents for people — finding products, comparing offers and completing purchases — within limits the person has approved.
- Vending-BenchVending-Bench, prop. n.: A test in which an AI agent runs a simulated vending-machine business for months — ordering stock, setting prices, paying fees — to see whether it stays sensible and profitable over a long time.
Sources
- Yao, S., Shinn, N., Razavi, P. & Narasimhan, K. (2024). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045. arxiv.org/abs/2406.12045
Mentioned in
pass@kpass@k, n.: A score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code., Vending-BenchVending-Bench, prop. n.: A test in which an AI agent runs a simulated vending-machine business for months — ordering stock, setting prices, paying fees — to see whether it stays sensible and profitable over a long time.