object detection
In plain wordsFinding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene.
The task of locating and classifying every instance of a set of object categories in an image, output as a list of bounding boxesbounding box, n.: The rectangle a vision system draws around an object it has found, stored as a handful of numbers giving the rectangle's position and size., each with a class label and a confidence score. Classical detectors slid a window across the image and classified each patch with hand-crafted features (the Viola–Jones face detectorface detection, n.: Finding where the faces are in a photo or video, without working out whose faces they are: the step that lets a camera focus on people or blur them out., 2001; histograms of oriented gradients, 2005; deformable part models, 2008). Deep detectors fall into two families: two-stage methods such as Faster R-CNNR-CNN, prop. n.: A family of object detectors that first pick out image regions likely to contain objects, then use a neural network to label each region and tighten its box. first propose candidate regions and then classify and refine them, while one-stage methods such as YOLOYOLO, prop. n.: A family of fast object-detection models that look at an image once and immediately output boxes and labels for everything they recognize, quick enough for live video., SSD and RetinaNet predict boxes densely in a single pass, trading some accuracy for speed. Most rely on anchor boxesanchor box, n.: A ready-made box shape, placed at many spots across an image, that an object detector stretches and shifts to fit real objects instead of drawing boxes from scratch. or anchor-free point predictions and remove duplicates with non-maximum suppressionnon-maximum suppression, n.: A clean-up step in object detection: when the system draws several overlapping boxes around the same object, it keeps the most confident one and throws away the near-duplicates.; transformer detectors such as DETRDETR, prop. n.: An object detector built on the transformer, the design behind modern chatbots, that outputs a short list of objects in one go instead of sorting through thousands of guessed boxes. predict a set of objects directly. Accuracy is measured by mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100. at given intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area. thresholds, typically on COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems.. Open-vocabulary detectorsopen-vocabulary detection, n.: Object detection where you type what to look for, such as 'red umbrella' or 'forklift', and the system finds it, even if it was never trained on that exact category. accept text prompts instead of a fixed list of classes.
The detector misses pedestrians half hidden behind parked vans, so we added occluded examples to the training set.
Origin
Detection grew out of template matching and pattern recognition in the 1960s–70s. Paul Viola and Michael Jones's real-time face detector (2001) and the PASCAL Visual Object Classes challenge (2005–2012) shaped the modern task and its benchmarks; R-CNN (2013) and YOLO (2015) opened the deep-learning era.
Usage note
Distinct from image classificationimage classification, n.: Teaching a computer to say what a picture shows by choosing a label, such as 'cat', 'pneumonia' or 'stop sign', from a fixed list of possibilities. (one label per image, no location) and image segmentationimage segmentation, n.: Dividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything. (pixel-accurate regions). Elsewhere in AI, 'detection' can mean any alarm, as in fraud or anomaly detectionanomaly detection, n.: Automatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network.; in vision it always implies localization.
Related
- bounding boxbounding box, n.: The rectangle a vision system draws around an object it has found, stored as a handful of numbers giving the rectangle's position and size.
- YOLOYOLO, prop. n.: A family of fast object-detection models that look at an image once and immediately output boxes and labels for everything they recognize, quick enough for live video.
- R-CNNR-CNN, prop. n.: A family of object detectors that first pick out image regions likely to contain objects, then use a neural network to label each region and tighten its box.
- DETRDETR, prop. n.: An object detector built on the transformer, the design behind modern chatbots, that outputs a short list of objects in one go instead of sorting through thousands of guessed boxes.
- mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100.
- non-maximum suppressionnon-maximum suppression, n.: A clean-up step in object detection: when the system draws several overlapping boxes around the same object, it keeps the most confident one and throws away the near-duplicates.
- COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems.
- open-vocabulary detectionopen-vocabulary detection, n.: Object detection where you type what to look for, such as 'red umbrella' or 'forklift', and the system finds it, even if it was never trained on that exact category.
- image classificationimage classification, n.: Teaching a computer to say what a picture shows by choosing a label, such as 'cat', 'pneumonia' or 'stop sign', from a fixed list of possibilities.
- image segmentationimage segmentation, n.: Dividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything.
Sources
- Girshick, R., Donahue, J., Darrell, T. & Malik, J. (2014). Rich Feature Hierarchies for Accurate Object Detection and Semantic Segmentation. CVPR 2014. arxiv.org/abs/1311.2524
- Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J. & Zisserman, A. (2010). The PASCAL Visual Object Classes (VOC) Challenge. International Journal of Computer Vision 88, 303–338.
Mentioned in
computer visioncomputer vision, n.: The part of AI that lets computers make sense of images and video: recognizing faces, reading signs, spotting tumors on scans or judging how far away a car is., image classificationimage classification, n.: Teaching a computer to say what a picture shows by choosing a label, such as 'cat', 'pneumonia' or 'stop sign', from a fixed list of possibilities., adversarial patchadversarial patch, n.: A printed sticker or picture designed so that, when it appears in a camera's view, an AI vision system misreads the scene — seeing a toaster, say, or missing a person., bounding boxbounding box, n.: The rectangle a vision system draws around an object it has found, stored as a handful of numbers giving the rectangle's position and size., COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems., face detectionface detection, n.: Finding where the faces are in a photo or video, without working out whose faces they are: the step that lets a camera focus on people or blur them out., focal lossfocal loss, n.: A way of scoring a model's mistakes during training that pays less attention to examples it already gets right easily, so that learning concentrates on the hard and rare cases., image segmentationimage segmentation, n.: Dividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything., instance segmentationinstance segmentation, n.: Finding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob., intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area., Kaiming HeKaiming He, prop. n.: A Chinese computer scientist whose ResNet design, with shortcut connections that let networks grow hundreds of layers deep, is among the most cited research in all of science., mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100., non-maximum suppressionnon-maximum suppression, n.: A clean-up step in object detection: when the system draws several overlapping boxes around the same object, it keeps the most confident one and throws away the near-duplicates., object trackingobject tracking, n.: Following objects from frame to frame in a video and keeping track of which is which, so a system knows that the car it sees now is the same car it saw a second ago., precision–recall curveprecision–recall curve, n.: A graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare., R-CNNR-CNN, prop. n.: A family of object detectors that first pick out image regions likely to contain objects, then use a neural network to label each region and tighten its box., YOLOYOLO, prop. n.: A family of fast object-detection models that look at an image once and immediately output boxes and labels for everything they recognize, quick enough for live video., anchor boxanchor box, n.: A ready-made box shape, placed at many spots across an image, that an object detector stretches and shifts to fit real objects instead of drawing boxes from scratch., DETRDETR, prop. n.: An object detector built on the transformer, the design behind modern chatbots, that outputs a short list of objects in one go instead of sorting through thousands of guessed boxes., ICCVICCV, prop. n.: A major international computer vision conference, held in odd-numbered years, where much of the research on how machines interpret images and video is first presented., open-vocabulary detectionopen-vocabulary detection, n.: Object detection where you type what to look for, such as 'red umbrella' or 'forklift', and the system finds it, even if it was never trained on that exact category., visual groundingvisual grounding, n.: Linking words to the exact part of an image they refer to, for example finding 'the man in the red hat on the left' and pointing to him.