visual grounding
In plain wordsLinking words to the exact part of an image they refer to, for example finding 'the man in the red hat on the left' and pointing to him.
The task of localizing the image region, as a bounding boxbounding box, n.: The rectangle a vision system draws around an object it has found, stored as a handful of numbers giving the rectangle's position and size., mask or point, that a natural-language expression refers to. Referring expression comprehension locates the single object described by a phrase such as 'the second mug from the left', which may depend on attributes, relations and context; phrase grounding localizes every noun phrase in a caption; referring segmentation outputs a mask. Benchmarks include RefCOCO, RefCOCO+ and RefCOCOg (2016) and Flickr30k Entities (2015). Specialized models such as MDETR (2021), GLIP (2022) and Grounding DINO (2023) fuse text and image features inside a detector; since 2023, multimodal language modelsmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type. ground by emitting coordinates as text or special tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens.. Grounding is central to GUI agentsGUI agent, n.: An AI agent that works through the buttons, menus and screens of ordinary software — on a desktop, phone or website — instead of through behind-the-scenes programming interfaces. and computer usecomputer use, n.: An AI's ability to work a computer through its normal screen and controls: it studies pictures of what is displayed and sends mouse and keyboard actions, so it can use ordinary apps that were built for people., where a model must map an instruction such as 'click the Submit button' to exact screen coordinates. It makes multimodal outputs checkable and helps reduce object hallucinationhallucination, n.: When an AI states something false or made up — a fake fact, quotation or source — fluently and confidently, as if it were true..
The agent understood the instruction but failed at grounding, clicking the search box instead of the address bar.
Origin
Linking language to image regions grew out of work on referring expressions and image–text alignment around 2014–2016, including Sahar Kazemzadeh and colleagues' ReferItGame (2014) and the RefCOCO datasets (2016). The phrase 'visual grounding' was in paper titles by 2016 (Akira Fukui and colleagues, EMNLP 2016) and became the common umbrella term in the late 2010s. GUI grounding emerged with screen-operating agents in 2023–2024.
Usage note
Narrower than the general groundinggrounding, n.: Connecting an AI's words to something real — trusted documents, a live search, images or the physical world — so that its answers rest on evidence rather than only on what it absorbed in training. of AI outputs in sources or in the world, and distinct from the philosophical symbol grounding problemsymbol grounding problem, n.: The puzzle of how words or symbols inside a machine can mean anything by themselves, rather than only pointing to more symbols — like a dictionary that defines every word using other words., although all three share the idea of tying symbols to what they refer to.
Related
- groundinggrounding, n.: Connecting an AI's words to something real — trusted documents, a live search, images or the physical world — so that its answers rest on evidence rather than only on what it absorbed in training.
- open-vocabulary detectionopen-vocabulary detection, n.: Object detection where you type what to look for, such as 'red umbrella' or 'forklift', and the system finds it, even if it was never trained on that exact category.
- visual question answeringvisual question answering, n.: Asking an AI a question about a picture, such as 'How many people are wearing hats?', and having it answer in words.
- vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence.
- GUI agentGUI agent, n.: An AI agent that works through the buttons, menus and screens of ordinary software — on a desktop, phone or website — instead of through behind-the-scenes programming interfaces.
- computer usecomputer use, n.: An AI's ability to work a computer through its normal screen and controls: it studies pictures of what is displayed and sends mouse and keyboard actions, so it can use ordinary apps that were built for people.
- object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene.
Sources
- Kazemzadeh, S., Ordonez, V., Matten, M. & Berg, T. L. (2014). ReferItGame: Referring to Objects in Photographs of Natural Scenes. EMNLP 2014.
- Yu, L., Poirson, P., Yang, S., Berg, A. C. & Berg, T. L. (2016). Modeling Context in Referring Expressions. ECCV 2016.
Mentioned in
groundinggrounding, n.: Connecting an AI's words to something real — trusted documents, a live search, images or the physical world — so that its answers rest on evidence rather than only on what it absorbed in training., OSWorldOSWorld, prop. n.: A test in which AI agents must use real desktop computers — clicking, typing and switching between apps such as spreadsheets, browsers and file managers — to finish everyday office tasks., visual question answeringvisual question answering, n.: Asking an AI a question about a picture, such as 'How many people are wearing hats?', and having it answer in words., GUI agentGUI agent, n.: An AI agent that works through the buttons, menus and screens of ordinary software — on a desktop, phone or website — instead of through behind-the-scenes programming interfaces., open-vocabulary detectionopen-vocabulary detection, n.: Object detection where you type what to look for, such as 'red umbrella' or 'forklift', and the system finds it, even if it was never trained on that exact category.