Skip to content
Last Dictionary
Plane
Vision, Speech & PerceptionDeep Learning

panoptic segmentation

nounAlso: panoptic quality, PQ, things and stuff

In plain wordsA complete labeling of an image in which every pixel gets a category, such as road or sky, and every countable object, such as each car or person, also gets its own identity.

  1. The task of assigning every pixel of an image both a semantic class and, for countable 'thing' classes, an instance identity, so that the output is a single non-overlapping partition of the whole image. It unifies semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another., which handles amorphous 'stuff' (sky, road, vegetation) but merges objects, and instance segmentationinstance segmentation, n.: Finding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob., which separates 'things' (people, cars) but ignores background and allows overlapping masks. Its metric, panoptic quality (PQ), matches a predicted and a ground-truth segment of the same class when their intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area. exceeds 0.5, a threshold that guarantees each match is unique, and equals segmentation quality (the mean IoU of matched segments) multiplied by recognition quality (an F1-style detection score). Early methods merged the outputs of separate semantic and instance networks (Panoptic FPN, 2019); later ones predict all segments as one set with a transformer (MaskFormer, 2021; Mask2Former, 2022). COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems. and driving datasets such as Cityscapes and Mapillary Vistas provide panoptic labels.

    The panoptic output gives the planner the drivable surface and a separate ID for every vehicle.

PQ=∑(p,g)∈TPIoU⁡(p,g)∣TP∣+12∣FP∣+12∣FN∣\text{PQ} = \frac{\sum_{(p,g)\in TP}\operatorname{IoU}(p,g)}{|TP| + \tfrac{1}{2}|FP| + \tfrac{1}{2}|FN|}

Origin

Defined by Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother and Piotr Dollár in 'Panoptic Segmentation' (arXiv January 2018; CVPR 2019), which also introduced the PQ metric; 'panoptic' is from Greek for 'all-seeing'. The COCO panoptic challenge began in 2018.

Related

See also
  • image segmentationimage segmentation, n.: Dividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything.
  • semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another.
  • instance segmentationinstance segmentation, n.: Finding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob.
  • intersection over unionintersection over union, n.: A score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area.
  • COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems.
  • DETRDETR, prop. n.: An object detector built on the transformer, the design behind modern chatbots, that outputs a short list of objects in one go instead of sorting through thousands of guessed boxes.

Sources

  1. Kirillov, A., He, K., Girshick, R., Rother, C. & Dollár, P. (2019). Panoptic Segmentation. CVPR 2019. arxiv.org/abs/1801.00868

Mentioned in

image segmentationimage segmentation, n.: Dividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything., instance segmentationinstance segmentation, n.: Finding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob., semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another., DETRDETR, prop. n.: An object detector built on the transformer, the design behind modern chatbots, that outputs a short list of objects in one go instead of sorting through thousands of guessed boxes.