depth estimation
In plain wordsWorking out how far away each point in a scene is from the camera, producing a 'depth map' that turns a flat picture into a sense of near and far.
The task of predicting, for each pixel, the distance from the camera to the visible surface, producing a depth map. Stereo methods find the same point in two horizontally offset images and convert the shift between them (the disparity d) into depth Z = fB/d, where f is the focal length and B the distance between the cameras; multi-view stereo generalizes this to many images. Active sensors (structured light, time-of-flight, lidarlidar, n.: A sensor that measures distances by firing laser pulses and timing their reflections, building a 3D picture of its surroundings out of millions of points; widely used on robots and self-driving cars.) measure depth directly. Monocular depth estimation infers depth from a single image using learned cues such as perspective, texture gradients, occlusion and the familiar sizes of objects; because absolute scale is ambiguous, models predict either relative (scale- and shift-invariant) depth or metric depth in meters, the latter requiring camera information or training on calibrated data. Milestones include multi-scale networks trained on indoor Kinect and outdoor lidar data (2014), self-supervised training from stereo pairs and video (2016–2017), training on mixtures of datasets (MiDaS, 2019) and large models trained on tens of millions of automatically labeled images, such as Depth Anything (2024). Uses include robotics, driver assistance, augmented reality, portrait-mode photo effects and 3D content creation.
Monocular depth is good enough for the background blur, but the robot still relies on its stereo camera.
Origin
Computational stereo was formulated in the 1970s, notably in David Marr and Tomaso Poggio's cooperative stereo algorithm (1976), drawing on photogrammetry. Deep monocular depth estimation began with David Eigen, Christian Puhrsch and Rob Fergus's multi-scale network (NeurIPS 2014).
Usage note
Relative depth maps look convincing but cannot be used directly for measurement; check whether a model outputs relative depth, metric depth or inverse depth (disparity).
Related
- 3D reconstruction3D reconstruction, n.: Building a 3D model of an object or place, its shape and often its colors, from photos, video or depth-sensor scans.
- lidarlidar, n.: A sensor that measures distances by firing laser pulses and timing their reflections, building a 3D picture of its surroundings out of millions of points; widely used on robots and self-driving cars.
- optical flowoptical flow, n.: How each part of a video picture seems to shift from one frame to the next, recorded as a tiny arrow per pixel giving the direction and distance moved.
- structure from motionstructure from motion, n.: Recovering both the 3D shape of a scene and the spot from which each photo was taken, using nothing but a collection of overlapping photographs.
- semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another.
- point cloudpoint cloud, n.: A 3D picture made of many dots floating in space, each marking a spot on a surface, as captured by a laser scanner or worked out from photos.
Sources
- Eigen, D., Puhrsch, C. & Fergus, R. (2014). Depth Map Prediction from a Single Image Using a Multi-Scale Deep Network. NeurIPS 2014.
- Yang, L., Kang, B., Huang, Z., Xu, X., Feng, J. & Zhao, H. (2024). Depth Anything: Unleashing the Power of Large-Scale Unlabeled Data. CVPR 2024. arxiv.org/abs/2401.10891
Mentioned in
computer visioncomputer vision, n.: The part of AI that lets computers make sense of images and video: recognizing faces, reading signs, spotting tumors on scans or judging how far away a car is., 3D reconstruction3D reconstruction, n.: Building a 3D model of an object or place, its shape and often its colors, from photos, video or depth-sensor scans., ControlNetControlNet, prop. n.: An add-on for image generators that makes them follow a guide image — a sketch, a stick-figure pose or a depth map — so the layout comes out as intended., lidarlidar, n.: A sensor that measures distances by firing laser pulses and timing their reflections, building a 3D picture of its surroundings out of millions of points; widely used on robots and self-driving cars., neural radiance fieldneural radiance field, n.: A way of capturing a real scene from a set of photos as a small neural network that can then show the scene from any new viewpoint, as if moving a camera through it., point cloudpoint cloud, n.: A 3D picture made of many dots floating in space, each marking a spot on a surface, as captured by a laser scanner or worked out from photos., semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another., DINODINO, prop. n.: A family of Meta vision models that learn to understand images from millions of unlabeled photos, by training one network to agree with a slowly updated copy of itself., optical flowoptical flow, n.: How each part of a video picture seems to shift from one frame to the next, recorded as a tiny arrow per pixel giving the direction and distance moved.