instance segmentation
In plain wordsFinding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob.
The task of detecting each individual object of a set of categories and producing a pixel mask for it, combining object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene. (separate instances) with semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another. (pixel accuracy). Unlike semantic segmentation, touching objects of the same class receive distinct masks and identities; unlike panoptic segmentationpanoptic segmentation, n.: A complete labeling of an image in which every pixel gets a category, such as road or sky, and every countable object, such as each car or person, also gets its own identity., it usually ignores amorphous background 'stuff' and may allow masks to overlap. Top-down methods detect a box and then segment inside it: Mask R-CNN (2017) added a small mask-predicting branch to Faster R-CNNR-CNN, prop. n.: A family of object detectors that first pick out image regions likely to contain objects, then use a neural network to label each region and tighten its box. and became the standard baseline. Single-stage and bottom-up alternatives predict masks directly or group pixels by learned embeddings (YOLACT, 2019, for real time; SOLO, 2020), and query-based transformers such as Mask2Former (2022) predict a set of masks much as DETRDETR, prop. n.: An object detector built on the transformer, the design behind modern chatbots, that outputs a short list of objects in one go instead of sorting through thousands of guessed boxes. predicts boxes. Evaluation uses mask APmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100. computed over mask IoU thresholds, typically on COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems. or the long-tailed LVIS dataset. Applications include robotic picking, cell counting in microscopy, image editing and autonomous driving.
Instance segmentation gives each apple its own mask, so the picker can plan a grasp for every fruit.
Origin
Bharath Hariharan, Pablo Arbeláez, Ross Girshick and Jitendra Malik's 'Simultaneous Detection and Segmentation' (ECCV 2014) framed the modern task; the phrase was already in titles that year, as in Nathan Silberman, David Sontag and Rob Fergus's 'Instance Segmentation of Indoor Scenes Using a Coverage Loss' (ECCV 2014). It became standard around 2015–2016 with the COCO segmentation challenge, and Mask R-CNN (2017) fixed the dominant approach.
Usage note
Some papers say 'object segmentation' for the same task; in video, 'video object segmentation' means following given objects' masks through the frames.
Related
- image segmentationimage segmentation, n.: Dividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything.
- semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another.
- panoptic segmentationpanoptic segmentation, n.: A complete labeling of an image in which every pixel gets a category, such as road or sky, and every countable object, such as each car or person, also gets its own identity.
- R-CNNR-CNN, prop. n.: A family of object detectors that first pick out image regions likely to contain objects, then use a neural network to label each region and tighten its box.
- object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene.
- Segment Anything ModelSegment Anything Model, prop. n.: A family of AI models from Meta that trace the exact outline of whatever you point to in a photo or video, whether by clicking on it, drawing a box around it or, in newer versions, naming it.
- semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another.
Sources
- Hariharan, B., Arbeláez, P., Girshick, R. & Malik, J. (2014). Simultaneous Detection and Segmentation. ECCV 2014.
- He, K., Gkioxari, G., Dollár, P. & Girshick, R. (2017). Mask R-CNN. ICCV 2017. arxiv.org/abs/1703.06870
Mentioned in
bounding boxbounding box, n.: The rectangle a vision system draws around an object it has found, stored as a handful of numbers giving the rectangle's position and size., COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems., image segmentationimage segmentation, n.: Dividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything., Kaiming HeKaiming He, prop. n.: A Chinese computer scientist whose ResNet design, with shortcut connections that let networks grow hundreds of layers deep, is among the most cited research in all of science., mean average precisionmean average precision, n.: The standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100., R-CNNR-CNN, prop. n.: A family of object detectors that first pick out image regions likely to contain objects, then use a neural network to label each region and tighten its box., Segment Anything ModelSegment Anything Model, prop. n.: A family of AI models from Meta that trace the exact outline of whatever you point to in a photo or video, whether by clicking on it, drawing a box around it or, in newer versions, naming it., semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another., YOLOYOLO, prop. n.: A family of fast object-detection models that look at an image once and immediately output boxes and labels for everything they recognize, quick enough for live video., panoptic segmentationpanoptic segmentation, n.: A complete labeling of an image in which every pixel gets a category, such as road or sky, and every countable object, such as each car or person, also gets its own identity.