U-Net
In plain wordsAn image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost.
An encoder–decoderencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output. convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects. that outputs a prediction for every pixel. Its encoder alternates convolutionsconvolution, n.: An operation that slides a small grid of weights across an image or signal, multiplying and adding at each spot, to measure how strongly a pattern appears at every location. with poolingpooling, n.: A step in an image network that shrinks its internal picture by summarizing each small patch with one number, usually the largest or the average, keeping what matters and discarding exact positions., so each stage sees a coarser but wider view of the image; its decoder mirrors the encoder, enlarging the feature mapsfeature map, n.: The grid of numbers produced when one learned pattern detector is scanned across an image, showing where, and how strongly, that pattern appears. stage by stage with up-convolutionstransposed convolution, n.: An operation that roughly reverses an image network's shrinking steps, enlarging a small grid of features into a bigger one, as when a model paints a full-size image.; and at each scale a skip connection concatenates the encoder's maps onto the decoder's, restoring the fine detail that downsampling discarded so that object boundaries come out sharp. Drawn as a diagram, the two halves form a U. It was created for biomedical image segmentationimage segmentation, n.: Dividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything., where labeled images are scarce, and relied on heavy data augmentationdata augmentation, n.: Creating extra training examples by changing existing ones in ways that keep their meaning — a flipped, cropped or recolored photo of a cat still shows a cat — so the model learns more robustly.; it became the standard segmentation architecture in medicine and beyond and, with added attention layers and timestep conditioning, the usual denoising network of diffusion modelsdiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. such as Stable DiffusionStable Diffusion, prop. n.: A widely used image-generating AI whose model files were released for anyone to download, so people could run it on their own computers and adapt it., though diffusion transformersdiffusion transformer, n.: A diffusion image or video generator whose core network is a transformer — the architecture behind chatbots — instead of the older image-specific design, which lets it improve more predictably as it grows. have increasingly replaced it since 2023.
The segmentation model is a standard U-Net with a ResNet-34 encoder.
Origin
Introduced by Olaf Ronneberger, Philipp Fischer and Thomas Brox of the University of Freiburg in 'U-Net: Convolutional Networks for Biomedical Image Segmentation' (arXiv, May 2015; MICCAI 2015), which won the ISBI 2015 cell-tracking challenge by a large margin. Jonathan Ho and colleagues' denoising diffusion models (2020) and later latent diffusion models adopted it as their denoiser.
Related
- image segmentationimage segmentation, n.: Dividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything.
- convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects.
- transposed convolutiontransposed convolution, n.: An operation that roughly reverses an image network's shrinking steps, enlarging a small grid of features into a bigger one, as when a model paints a full-size image.
- diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
- encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output.
- residual connectionresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable.
Sources
- Ronneberger, O., Fischer, P. & Brox, T. (2015). U-Net: Convolutional Networks for Biomedical Image Segmentation. MICCAI 2015; arXiv:1505.04597. arxiv.org/abs/1505.04597
Mentioned in
diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static., Stable DiffusionStable Diffusion, prop. n.: A widely used image-generating AI whose model files were released for anyone to download, so people could run it on their own computers and adapt it., convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects., diffusion transformerdiffusion transformer, n.: A diffusion image or video generator whose core network is a transformer — the architecture behind chatbots — instead of the older image-specific design, which lets it improve more predictably as it grows., encoder–decoder architectureencoder–decoder architecture, n.: A two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output., feature mapfeature map, n.: The grid of numbers produced when one learned pattern detector is scanned across an image, showing where, and how strongly, that pattern appears., image segmentationimage segmentation, n.: Dividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything., image-to-image translationimage-to-image translation, n.: Turning one picture into another version of itself — a sketch into a photo, a day scene into night, a horse into a zebra — while keeping its layout., residual connectionresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable., semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another., denoising diffusion probabilistic modeldenoising diffusion probabilistic model, n.: The 2020 recipe that made diffusion image generators work well: add noise to pictures over a thousand small steps, then train a network to guess the noise so it can be removed., DenseNetDenseNet, prop. n.: An image-recognition network in which every layer receives the outputs of all the layers before it, so features are reused rather than relearned., diffusion policydiffusion policy, n.: A way of training robots in which the same kind of AI that generates images instead generates the robot's next movements, letting it copy demonstrations even when there are several good ways to do a task., fully convolutional networkfully convolutional network, n.: An image network made only of scanning-window layers, with no fixed-size final layers, so it can take pictures of any size and label every pixel., transposed convolutiontransposed convolution, n.: An operation that roughly reverses an image network's shrinking steps, enlarging a small grid of features into a bigger one, as when a model paints a full-size image.