semantic segmentation
In plain wordsColoring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another.
The task of assigning a class label from a fixed set to every pixel of an image, producing a label map the size of the input. Objects of the same class merge into one region, so two adjacent people become a single 'person' area; separating them is instance segmentationinstance segmentation, n.: Finding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob.. Semantic segmentation covers both countable 'things' and amorphous 'stuff' such as road, grass and sky, which makes it the basis of scene understanding in driving and robotics. Fully convolutional networks (2014) adapted classification networks to output dense predictions; later architectures recovered spatial detail and context with encoder–decoder skip connections (U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost.), dilated (atrous) convolutions and multi-scale pooling (the DeepLab family, 2014–2018), and attention (SegFormer, Mask2Former). Self-supervised backbonesbackbone, n.: The main, usually pretrained part of a vision model that turns an image into useful internal features; smaller task-specific parts are attached to it, like limbs to a spine. such as DINOv2DINO, prop. n.: A family of Meta vision models that learn to understand images from millions of unlabeled photos, by training one network to agree with a slowly updated copy of itself. give strong results with only a light decoder. Standard benchmarks include PASCAL VOC, Cityscapes (urban driving, 2016) and ADE20K (150 classes, 2017); the usual metric is mean intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area. (mIoU) across classes.
Semantic segmentation marks the drivable area; a separate instance model counts the pedestrians.
Origin
Region labeling guided by knowledge of object classes was explored in 1970s scene-analysis systems. The phrase 'semantic segmentation' spread in the late 2000s with work on labeling pixels by object class, and the fully convolutional networks of Jonathan Long, Evan Shelhamer and Trevor Darrell (2014) made end-to-end deep models the standard approach.
Usage note
Medical imaging often calls the same task simply 'segmentation' (of organs, lesions or cell types). 'Dense prediction' is the wider family of per-pixel tasks that also includes depth estimationdepth estimation, n.: Working out how far away each point in a scene is from the camera, producing a 'depth map' that turns a flat picture into a sense of near and far. and optical flowoptical flow, n.: How each part of a video picture seems to shift from one frame to the next, recorded as a tiny arrow per pixel giving the direction and distance moved..
Related
- image segmentationimage segmentation, n.: Dividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything.
- instance segmentationinstance segmentation, n.: Finding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob.
- panoptic segmentationpanoptic segmentation, n.: A complete labeling of an image in which every pixel gets a category, such as road or sky, and every countable object, such as each car or person, also gets its own identity.
- U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost.
- intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area.
- convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects.
- instance segmentationinstance segmentation, n.: Finding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob.
Sources
- Long, J., Shelhamer, E. & Darrell, T. (2015). Fully Convolutional Networks for Semantic Segmentation. CVPR 2015. arxiv.org/abs/1411.4038
- Chen, L.-C., Papandreou, G., Kokkinos, I., Murphy, K. & Yuille, A. L. (2018). DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFs. IEEE TPAMI 40(4), 834–848.
Mentioned in
depth estimationdepth estimation, n.: Working out how far away each point in a scene is from the camera, producing a 'depth map' that turns a flat picture into a sense of near and far., image segmentationimage segmentation, n.: Dividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything., instance segmentationinstance segmentation, n.: Finding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob., intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area., dilated convolutiondilated convolution, n.: A version of the scanning-window operation that spreads its weights apart with gaps, so it can take in a wider area of an image or sound at no extra cost., fully convolutional networkfully convolutional network, n.: An image network made only of scanning-window layers, with no fixed-size final layers, so it can take pictures of any size and label every pixel., panoptic segmentationpanoptic segmentation, n.: A complete labeling of an image in which every pixel gets a category, such as road or sky, and every countable object, such as each car or person, also gets its own identity.