image-to-image translation
In plain wordsTurning one picture into another version of itself — a sketch into a photo, a day scene into night, a horse into a zebra — while keeping its layout.
The task of transforming an image so that it takes on the look of a different domain while its layout and content stay in place, such as turning day scenes into night, satellite images into maps, sketches into product photos or summer landscapes into winter ones. With paired examples, a conditional generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people. can learn the transformation directly, as in pix2pix, which combined a U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost. generator, a patch-based discriminator and an L1 reconstruction loss. Without pairs, CycleGAN trained two generators in opposite directions and added a cycle-consistency loss requiring that translating an image there and back recovers the original. Diffusion modelsdiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. now dominate: SDEdit-style 'img2img' adds a chosen amount of noise to an input and denoises it under a new prompt, trading faithfulness against realism, while ControlNetControlNet, prop. n.: An add-on for image generators that makes them follow a guide image — a sketch, a stick-figure pose or a depth map — so the layout comes out as intended. and instruction-based image editinginstruction-based image editing, n.: Changing a picture by telling an AI what to change — 'make it night', 'remove the car', 'give her a red coat' — instead of using editing tools by hand. give finer control.
We used image-to-image translation to turn the simulator's flat renders into photorealistic training frames.
Origin
Popularized by Phillip Isola, Jun-Yan Zhu, Tinghui Zhou and Alexei Efros's 'Image-to-Image Translation with Conditional Adversarial Networks' (posted November 2016; CVPR 2017), whose pix2pix code spread widely among artists. CycleGAN, by Zhu, Taesung Park, Isola and Efros, followed in March 2017, and SDEdit (Chenlin Meng and colleagues, 2021) carried the idea into diffusion models.
Usage note
'Image Analogies' (Aaron Hertzmann and colleagues, SIGGRAPH 2001) is a classic non-neural precursor that learned filters from one before-and-after example.
Related
- generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people.
- style transferstyle transfer, n.: Repainting a photo in the style of an artwork — your street as Van Gogh might have painted it — by keeping the photo's content and borrowing the painting's colors, strokes and textures.
- ControlNetControlNet, prop. n.: An add-on for image generators that makes them follow a guide image — a sketch, a stick-figure pose or a depth map — so the layout comes out as intended.
- instruction-based image editinginstruction-based image editing, n.: Changing a picture by telling an AI what to change — 'make it night', 'remove the car', 'give her a red coat' — instead of using editing tools by hand.
- inpaintinginpainting, n.: Filling in a missing or erased part of a picture so it blends seamlessly with the rest — removing a stranger from a photo, or repairing a damaged one.
- super-resolutionsuper-resolution, n.: Making a small or blurry image larger and sharper by having AI fill in plausible fine detail, rather than just stretching the pixels.
- U-NetU-Net, prop. n.: An image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost.
Sources
- Isola, P., Zhu, J.-Y., Zhou, T. & Efros, A. A. (2017). Image-to-Image Translation with Conditional Adversarial Networks. CVPR 2017. arXiv:1611.07004. arxiv.org/abs/1611.07004
- Zhu, J.-Y., Park, T., Isola, P. & Efros, A. A. (2017). Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks. ICCV 2017. arXiv:1703.10593. arxiv.org/abs/1703.10593
- Meng, C. et al. (2022). SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations. ICLR 2022. arXiv:2108.01073. arxiv.org/abs/2108.01073
Mentioned in
ControlNetControlNet, prop. n.: An add-on for image generators that makes them follow a guide image — a sketch, a stick-figure pose or a depth map — so the layout comes out as intended., generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people., inpaintinginpainting, n.: Filling in a missing or erased part of a picture so it blends seamlessly with the rest — removing a stranger from a photo, or repairing a damaged one., instruction-based image editinginstruction-based image editing, n.: Changing a picture by telling an AI what to change — 'make it night', 'remove the car', 'give her a red coat' — instead of using editing tools by hand., style transferstyle transfer, n.: Repainting a photo in the style of an artwork — your street as Van Gogh might have painted it — by keeping the photo's content and borrowing the painting's colors, strokes and textures., StyleGANStyleGAN, prop. n.: An image-generating network from NVIDIA, famous for photorealistic faces of people who do not exist, that sets coarse traits such as head angle and face shape separately from fine details such as coloring and skin texture., super-resolutionsuper-resolution, n.: Making a small or blurry image larger and sharper by having AI fill in plausible fine detail, rather than just stretching the pixels.