GUI agent
In plain wordsAn AI agent that works through the buttons, menus and screens of ordinary software — on a desktop, phone or website — instead of through behind-the-scenes programming interfaces.
An AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done. that perceives and operates graphical user interfaces — desktop applications, mobile apps and web pages — by grounding natural-language goals in on-screen elements and executing interface actions such as taps, clicks, swipes and text entry. GUI agents rely on vision-language modelsvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence. to locate elements in screenshots (visual groundingvisual grounding, n.: Linking words to the exact part of an image they refer to, for example finding 'the man in the red hat on the left' and pointing to him.), sometimes aided by accessibility trees, view hierarchies or 'set-of-marks' annotations that number the clickable regions. The research area spans specialized grounding models, end-to-end 'native' agent models trained on large collections of interface trajectories, and benchmarks for desktop (OSWorldOSWorld, prop. n.: A test in which AI agents must use real desktop computers — clicking, typing and switching between apps such as spreadsheets, browsers and file managers — to finish everyday office tasks.), web (WebArenaWebArena, prop. n.: A test world of realistic, self-hosted websites — a code-hosting site, an online shop, a forum and more — in which AI agents must complete tasks such as finding orders or editing posts.) and mobile (AndroidWorld) tasks. Main difficulties are precise localization on high-resolution screens, long action sequences, recovery from unexpected dialogs and pop-ups, and safety around irreversible actions.
Our GUI agent fails mostly at grounding: it knows to click Export but misses the button by a few pixels.
Origin
Builds on interface automation, robotic process automation and reinforcement-learning agents for web tasks (World of Bits, 2017). The term spread in 2023–2024 with vision-language models able to read screens, such as Tsinghua and Zhipu AI's CogAgent (December 2023) and Apple's Ferret-UI (April 2024), and with surveys of 'GUI agents'; Anthropic's computer usecomputer use, n.: An AI's ability to work a computer through its normal screen and controls: it studies pictures of what is displayed and sends mouse and keyboard actions, so it can use ordinary apps that were built for people. release (October 2024) brought the capability to the mainstream.
Related
- computer usecomputer use, n.: An AI's ability to work a computer through its normal screen and controls: it studies pictures of what is displayed and sends mouse and keyboard actions, so it can use ordinary apps that were built for people.
- browser agentbrowser agent, n.: An AI agent that uses a web browser on your behalf — searching, clicking links, filling in forms and reading pages — to complete tasks such as booking, shopping or gathering information.
- OSWorldOSWorld, prop. n.: A test in which AI agents must use real desktop computers — clicking, typing and switching between apps such as spreadsheets, browsers and file managers — to finish everyday office tasks.
- vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence.
- visual groundingvisual grounding, n.: Linking words to the exact part of an image they refer to, for example finding 'the man in the red hat on the left' and pointing to him.
- AI agentAI agent, n.: An AI system that does not just answer but acts: given a goal, it decides on steps, uses tools such as a browser or code editor, checks the results and keeps going until the job is done.
Sources
- Hong, W. et al. (2023). CogAgent: A Visual Language Model for GUI Agents. arXiv:2312.08914 (CVPR 2024).
Mentioned in
computer usecomputer use, n.: An AI's ability to work a computer through its normal screen and controls: it studies pictures of what is displayed and sends mouse and keyboard actions, so it can use ordinary apps that were built for people., browser agentbrowser agent, n.: An AI agent that uses a web browser on your behalf — searching, clicking links, filling in forms and reading pages — to complete tasks such as booking, shopping or gathering information., ByteDanceByteDance, prop. n.: The Chinese company that owns TikTok, which also runs Doubao, China's most popular AI chatbot, and builds image and video generators., OSWorldOSWorld, prop. n.: A test in which AI agents must use real desktop computers — clicking, typing and switching between apps such as spreadsheets, browsers and file managers — to finish everyday office tasks., visual groundingvisual grounding, n.: Linking words to the exact part of an image they refer to, for example finding 'the man in the red hat on the left' and pointing to him.