image segmentation
In plain wordsDividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything.
The task of dividing an image into regions by assigning every pixel to a segment or class label, producing masks rather than boxes. Three main formulations are distinguished: semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another. labels every pixel with a class without separating individual objects; instance segmentationinstance segmentation, n.: Finding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob. outlines each object separately; and panoptic segmentationpanoptic segmentation, n.: A complete labeling of an image in which every pixel gets a category, such as road or sky, and every countable object, such as each car or person, also gets its own identity. combines the two. Interactive or promptable segmentation, as in the Segment Anything ModelSegment Anything Model, prop. n.: A family of AI models from Meta that trace the exact outline of whatever you point to in a photo or video, whether by clicking on it, drawing a box around it or, in newer versions, naming it., produces a mask for whatever a user indicates with clicks, a box or text. Classical methods grouped pixels by intensity, color and texture: thresholding, region growing, the watershed transform, active contours ('snakes', 1988), normalized cuts (2000) and graph cuts. Deep methods took over with fully convolutional networks (2014) and encoder–decoder designs such as U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost. (2015), followed by Mask R-CNN and transformer-based mask classifiers such as Mask2Former (2022). Quality is measured with intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area. or the Dice coefficient. Segmentation underpins medical imaging (outlining tumors and organs), photo editing, autonomous driving, agriculture and satellite mapping.
The segmentation masks show the robot exactly where one tangled cable ends and the next begins.
Origin
Segmentation was a central problem of early image analysis: thresholding, region-growing and edge-linking methods were developed from the 1960s–70s, and Nobuyuki Otsu's 1979 threshold-selection method became a classic. After earlier patch-based convolutional approaches (2012–2013), Jonathan Long, Evan Shelhamer and Trevor Darrell's fully convolutional networks (arXiv November 2014), trained end to end to label every pixel, set the template for deep segmentation.
Usage note
A mask is a binary image marking the pixels of one segment. In speech and text processing, 'segmentation' means cutting a signal or string into units (utterances, words) and is unrelated.
Related
- semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another.
- instance segmentationinstance segmentation, n.: Finding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob.
- panoptic segmentationpanoptic segmentation, n.: A complete labeling of an image in which every pixel gets a category, such as road or sky, and every countable object, such as each car or person, also gets its own identity.
- Segment Anything ModelSegment Anything Model, prop. n.: A family of AI models from Meta that trace the exact outline of whatever you point to in a photo or video, whether by clicking on it, drawing a box around it or, in newer versions, naming it.
- U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost.
- intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area.
- object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene.
Sources
- Long, J., Shelhamer, E. & Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. CVPR 2015. arxiv.org/abs/1411.4038
- Ronneberger, O., Fischer, P. & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI 2015. arxiv.org/abs/1505.04597
Mentioned in
computer visioncomputer vision, n.: The part of AI that lets computers make sense of images and video: recognizing faces, reading signs, spotting tumors on scans or judging how far away a car is., image classificationimage classification, n.: Teaching a computer to say what a picture shows by choosing a label, such as 'cat', 'pneumonia' or 'stop sign', from a fixed list of possibilities., object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene., instance segmentationinstance segmentation, n.: Finding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob., Segment Anything ModelSegment Anything Model, prop. n.: A family of AI models from Meta that trace the exact outline of whatever you point to in a photo or video, whether by clicking on it, drawing a box around it or, in newer versions, naming it., semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another., U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost., fully convolutional networkfully convolutional network, n.: An image network made only of scanning-window layers, with no fixed-size final layers, so it can take pictures of any size and label every pixel., panoptic segmentationpanoptic segmentation, n.: A complete labeling of an image in which every pixel gets a category, such as road or sky, and every countable object, such as each car or person, also gets its own identity.