visual question answering
In plain wordsAsking an AI a question about a picture, such as 'How many people are wearing hats?', and having it answer in words.
The task of answering a natural-language question about an image or video, which requires combining visual recognition with language understanding and sometimes counting, reading text, spatial reasoning or outside knowledge. Early systems fused a CNNconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects. image representation with a recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood. encoding of the question and classified over a fixed set of frequent answers; attention over image regions and transformer-based vision–language pretraining followed. Benchmark design proved critical: on the original VQA dataset, models could score well from language priors alone (answering 'yes' or 'tennis' without looking), so VQA v2 (2017) added, for each question, a second, similar-looking image on which the correct answer differs. Specialized datasets test compositional reasoning (GQA), outside knowledge (OK-VQA), text in images (TextVQA), photos taken by blind people (VizWiz), documents (DocVQA) and charts (ChartQA). Since 2023, vision-language modelsvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence. answer free-form, and VQA serves mainly as an evaluation format; the standard VQA accuracy gives full credit to an answer that at least three of ten human annotators gave.
On VizWiz the model often answers confidently when the photo is too blurry and the right reply is 'unanswerable'.
Origin
Mateusz Malinowski and Mario Fritz's DAQUAR dataset (2014) framed question answering about real-world images; the name and the large benchmark came from Stanislaw Antol, Aishwarya Agrawal, Devi Parikh and colleagues' 'VQA: Visual Question Answering' (ICCV 2015). Jeffrey Bigham and colleagues' VizWiz app (2010) had earlier crowdsourced answers to blind users' photo questions.
Usage note
The official metric averages the formula over subsets of the ten human answers. High VQA scores can reflect dataset biases rather than visual understanding, a recurring theme in multimodal evaluation.
Related
- vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence.
- image captioningimage captioning, n.: Having a computer write a short description of a picture, such as 'a dog catching a frisbee in a park', to help blind people, search engines and other AI systems.
- visual groundingvisual grounding, n.: Linking words to the exact part of an image they refer to, for example finding 'the man in the red hat on the left' and pointing to him.
- question answeringquestion answering, n.: Systems that reply to people's questions with the right answer, whether by finding it in a passage they are given, searching a database or the web, or drawing on what they already know.
- document understandingdocument understanding, n.: Using AI to read documents such as invoices, forms, contracts and scientific papers the way a person does, taking in layout, tables and charts as well as words, to pull out the information needed.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
Sources
- Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L. & Parikh, D. (2015). VQA: Visual Question Answering. ICCV 2015.
- Goyal, Y., Khot, T., Summers-Stay, D., Batra, D. & Parikh, D. (2017). Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering. CVPR 2017.
Mentioned in
document understandingdocument understanding, n.: Using AI to read documents such as invoices, forms, contracts and scientific papers the way a person does, taking in layout, tables and charts as well as words, to pull out the information needed., image captioningimage captioning, n.: Having a computer write a short description of a picture, such as 'a dog catching a frisbee in a park', to help blind people, search engines and other AI systems., MMMUMMMU, prop. n.: A test for AI models that understand pictures as well as text: 11,500 college-level questions across 30 subjects, each built around one or more images such as a diagram, a musical score or a chemical structure., multimodal AImultimodal AI, n.: AI that can take in or produce more than one kind of information, such as text, images, sound and video, so you can show it a photo and ask questions about it., multimodal large language modelmultimodal large language model, n.: An AI language model that can also take in — and sometimes produce — pictures, sound or video, so you can show it a photo or speak to it, not just type., question answeringquestion answering, n.: Systems that reply to people's questions with the right answer, whether by finding it in a passage they are given, searching a database or the web, or drawing on what they already know., vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence., visual groundingvisual grounding, n.: Linking words to the exact part of an image they refer to, for example finding 'the man in the red hat on the left' and pointing to him.