WebArena
In plain wordsA test world of realistic, self-hosted websites — a code-hosting site, an online shop, a forum and more — in which AI agents must complete tasks such as finding orders or editing posts.
A benchmark and test environment for autonomous browser agentsbrowser agent, n.: An AI agent that uses a web browser on your behalf — searching, clicking links, filling in forms and reading pages — to complete tasks such as booking, shopping or gathering information. made of working copies of open-source web applications that run locally — a code-hosting platform, an online store, a discussion forum and a content-management system — supplemented by utilities such as a map and a wiki. Its 812 tasks are multi-step instructions in natural language, and a program decides success by examining the final state of the site or the agent's answer, so any valid route to the goal counts. Because the sites run locally from container images, experiments are repeatable and unaffected by changes to the live web. When it appeared, humans completed 78.24% of the tasks and the strongest agent, built on GPT-4, 14.41%. Extensions include VisualWebArena (2024), which requires interpreting images on pages; by 2025, leading agents exceeded 60% on the original benchmark.
On WebArena the agent completes shopping tasks but fails most of the GitLab ones.
Origin
Introduced by Shuyan Zhou, Frank F. Xu, Hao Zhu, Xuhui Zhou and colleagues at Carnegie Mellon University in July 2023, and published at ICLR 2024.
Usage note
Some tasks have ambiguous instructions or brittle checkers, so reported success rates carry a few points of measurement error.
Related
- browser agentbrowser agent, n.: An AI agent that uses a web browser on your behalf — searching, clicking links, filling in forms and reading pages — to complete tasks such as booking, shopping or gathering information.
- computer usecomputer use, n.: An AI's ability to work a computer through its normal screen and controls: it studies pictures of what is displayed and sends mouse and keyboard actions, so it can use ordinary apps that were built for people.
- OSWorldOSWorld, prop. n.: A test in which AI agents must use real desktop computers — clicking, typing and switching between apps such as spreadsheets, browsers and file managers — to finish everyday office tasks.
- GAIAGAIA, prop. n.: A test of AI assistants made of real-world questions that people can answer with a browser and some patience, but that require an AI to search, read files and combine facts.
- AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
- BrowseCompBrowseComp, prop. n.: A test of AI web-research agents: 1,266 puzzle-like questions whose answers take a long, persistent online hunt to uncover, though each answer is short and simple to confirm.
Sources
- Zhou, S., Xu, F. F., Zhu, H. et al. (2024). WebArena: A Realistic Web Environment for Building Autonomous Agents. ICLR 2024. arxiv.org/abs/2307.13854
Mentioned in
browser agentbrowser agent, n.: An AI agent that uses a web browser on your behalf — searching, clicking links, filling in forms and reading pages — to complete tasks such as booking, shopping or gathering information., GAIAGAIA, prop. n.: A test of AI assistants made of real-world questions that people can answer with a browser and some patience, but that require an AI to search, read files and combine facts., OSWorldOSWorld, prop. n.: A test in which AI agents must use real desktop computers — clicking, typing and switching between apps such as spreadsheets, browsers and file managers — to finish everyday office tasks., GUI agentGUI agent, n.: An AI agent that works through the buttons, menus and screens of ordinary software — on a desktop, phone or website — instead of through behind-the-scenes programming interfaces.