optical flow
In plain wordsHow each part of a video picture seems to shift from one frame to the next, recorded as a tiny arrow per pixel giving the direction and distance moved.
The per-pixel motion field between two consecutive images: a displacement vector (u, v) for every pixel saying where the local brightness pattern appears to move, which may differ from the true motion of surfaces. Most methods assume brightness constancy, that a moving point keeps its intensity, which yields one linear equation per pixel in two unknowns (see formula, where I_x, I_y and I_t are image derivatives); the missing constraint, known as the aperture problem, is supplied by smoothness assumptions. Berthold Horn and Brian Schunck imposed smoothness across the whole image, while the Lucas–Kanade method assumes constant motion within small windows and solves by least squares; both appeared in 1981. Deep networks took over from FlowNet (2015) onward; RAFT (2020) compares all pairs of pixels in a correlation volume and refines the flow iteratively. Flow is evaluated by end-point error on benchmarks such as MPI Sintel and KITTI. It is used for video stabilization and compression, frame interpolation and slow motion, action recognitionaction recognition, n.: Teaching computers to recognize what people are doing in a video, such as running, waving, cooking or falling, rather than just which objects appear in it., object trackingobject tracking, n.: Following objects from frame to frame in a video and keeping track of which is which, so a system knows that the car it sees now is the same car it saw a second ago., visual odometryodometry, n.: Estimating how far and in which direction a robot or vehicle has moved by adding up its own motion, such as wheel turns or how camera images shift, like counting your steps. and robot navigation.
We warp the previous frame with the optical flow to stop the stylized video from flickering.
Origin
The psychologist James J. Gibson introduced the concept in the 1940s, for the flowing pattern of the visual field during movement. Motion estimation from image sequences was studied in computer vision in the 1970s, and Berthold Horn and Brian Schunck's 'Determining Optical Flow' and Bruce Lucas and Takeo Kanade's image-registration method, both published in 1981, founded its standard computation.
Usage note
Optical flow is the motion of image brightness, which can differ from true scene motion: a uniform sphere spinning in place produces no flow, and a moving light source produces flow without any moving object.
Related
- object trackingobject tracking, n.: Following objects from frame to frame in a video and keeping track of which is which, so a system knows that the car it sees now is the same car it saw a second ago.
- action recognitionaction recognition, n.: Teaching computers to recognize what people are doing in a video, such as running, waving, cooking or falling, rather than just which objects appear in it.
- depth estimationdepth estimation, n.: Working out how far away each point in a scene is from the camera, producing a 'depth map' that turns a flat picture into a sense of near and far.
- odometryodometry, n.: Estimating how far and in which direction a robot or vehicle has moved by adding up its own motion, such as wheel turns or how camera images shift, like counting your steps.
- computer visioncomputer vision, n.: The part of AI that lets computers make sense of images and video: recognizing faces, reading signs, spotting tumors on scans or judging how far away a car is.
Sources
- Horn, B. K. P. & Schunck, B. G. (1981). Determining Optical Flow. Artificial Intelligence 17, 185–203.
- Lucas, B. D. & Kanade, T. (1981). An Iterative Image Registration Technique with an Application to Stereo Vision. Proceedings of IJCAI 1981.
- Teed, Z. & Deng, J. (2020). RAFT: Recurrent All-Pairs Field Transforms for Optical Flow. ECCV 2020.
Mentioned in
computer visioncomputer vision, n.: The part of AI that lets computers make sense of images and video: recognizing faces, reading signs, spotting tumors on scans or judging how far away a car is., depth estimationdepth estimation, n.: Working out how far away each point in a scene is from the camera, producing a 'depth map' that turns a flat picture into a sense of near and far., object trackingobject tracking, n.: Following objects from frame to frame in a video and keeping track of which is which, so a system knows that the car it sees now is the same car it saw a second ago., semantic segmentationsemantic segmentation, n.: Coloring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another., action recognitionaction recognition, n.: Teaching computers to recognize what people are doing in a video, such as running, waving, cooking or falling, rather than just which objects appear in it., odometryodometry, n.: Estimating how far and in which direction a robot or vehicle has moved by adding up its own motion, such as wheel turns or how camera images shift, like counting your steps.