GAIA

A benchmark of 466 questions for general-purpose AI assistants, with the answers to 300 withheld to run a leaderboard. Answering one means chaining everyday skills — searching and browsing the web, using toolstool use, n.: An AI model's ability to reach beyond its own knowledge by operating outside tools — searching the web, running code, using a calculator or an app — and folding the results into its work., reading images or attached files such as spreadsheets, and reasoning over what turns up — and each question has a short factual answer, graded by near-exact matchexact match, n.: The simplest way to grade an AI's answer: it counts as right only if it is identical to the reference answer, after minor tidying such as ignoring capital letters., and a difficulty level from 1 to 3 set by how many steps and tools it takes. Unlike exam-style benchmarks that keep raising the bar for human experts, its questions are easy for people but hard for AI, on the argument that matching an ordinary person's reliability on such tasks says more about general usefulness than superhuman exam scores. In the original tests people answered 92% correctly and GPT-4 with plugins 15%. GAIA became a standard test for deep researchdeep research, n.: An AI feature that spends minutes rather than seconds on a question — planning, running dozens of web searches, reading many sources — and then writes a long report with citations. and general agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. systems.

Most of our GAIA failures come from misreading the attached spreadsheet, not from search.