Skip to content
Last Dictionary
Plane
Vision, Speech & PerceptionDeep LearningLanguage Models & Transformers

open-vocabulary detection

nounAlso: open-vocabulary object detection, OVD, open-set detection, zero-shot object detection, text-prompted detection, Grounding DINO, OWL-ViT, YOLO-World

In plain wordsObject detection where you type what to look for, such as 'red umbrella' or 'forklift', and the system finds it, even if it was never trained on that exact category.

  1. Object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene. in which the categories are supplied at inference time, usually as free text, instead of being fixed by the training labels. Models align region features with language embeddingsembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. so that any phrase can act as a classifier. Early systems learned this visual–semantic alignment from image–caption pairs or distilled it from CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image. (ViLD, 2021); later ones were pretrained on large mixtures of detection, grounding and caption data (GLIP and OWL-ViT, 2022; Grounding DINO, 2023; YOLO-World, 2024). Grounding DINO, for example, reported 52.5 AP on COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems. without any COCO training data. Evaluations separate 'base' classes seen with box labels from 'novel' classes known only through text. The idea extends to segmentation, as in the promptable concept segmentation of SAM 3Segment Anything Model, prop. n.: A family of AI models from Meta that trace the exact outline of whatever you point to in a photo or video, whether by clicking on it, drawing a box around it or, in newer versions, naming it. (2025), and shades into visual groundingvisual grounding, n.: Linking words to the exact part of an image they refer to, for example finding 'the man in the red hat on the left' and pointing to him., which locates one referred object rather than every instance of a category.

    With an open-vocabulary detector we added 'hard hat' and 'hi-vis vest' to the site monitor without labeling a single new image.

Origin

The task was named and formalized by Alireza Zareian, Kevin Dela Rosa, Derek Hao Hu and Shih-Fu Chang in 'Open-Vocabulary Object Detection Using Captions' (arXiv November 2020; CVPR 2021), building on zero-shot detection work from 2018; the release of CLIP in 2021 accelerated the field.

Usage note

'Open-set detection' can mean something different: flagging objects of unseen classes as 'unknown' rather than naming them. Some open-vocabulary models, including Grounding DINO, nonetheless describe themselves as open-set.

Related

See also
  • object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene.
  • CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image.
  • visual groundingvisual grounding, n.: Linking words to the exact part of an image they refer to, for example finding 'the man in the red hat on the left' and pointing to him.
  • zero-shot learningzero-shot learning, n.: Doing a task without being shown a single example of it — say, an AI sorting reviews into positive and negative just from being told to, or recognizing an animal it never saw in training.
  • vision-language modelvision-language model, n.: An AI model that understands images and text together — it can describe a photo, answer questions about a chart or find the picture that matches a sentence.
  • Segment Anything ModelSegment Anything Model, prop. n.: A family of AI models from Meta that trace the exact outline of whatever you point to in a photo or video, whether by clicking on it, drawing a box around it or, in newer versions, naming it.

Sources

  1. Zareian, A., Dela Rosa, K., Hu, D. H. & Chang, S.-F. (2021). Open-Vocabulary Object Detection Using Captions. CVPR 2021. arxiv.org/abs/2011.10678
  2. Liu, S. et al. (2023). Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection. arXiv:2303.05499. arxiv.org/abs/2303.05499

Mentioned in

object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene., CLIPCLIP, prop. n.: An OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image., Segment Anything ModelSegment Anything Model, prop. n.: A family of AI models from Meta that trace the exact outline of whatever you point to in a photo or video, whether by clicking on it, drawing a box around it or, in newer versions, naming it., DETRDETR, prop. n.: An object detector built on the transformer, the design behind modern chatbots, that outputs a short list of objects in one go instead of sorting through thousands of guessed boxes., visual groundingvisual grounding, n.: Linking words to the exact part of an image they refer to, for example finding 'the man in the red hat on the left' and pointing to him.