pose estimation
In plain wordsWorking out the position and orientation of something from images: where a person's joints are and how their body is posed, or exactly how an object or camera is placed and turned in space.
- 1
Human pose
Estimating the configuration of a human body from images or video, as 2D image locations of joints (wrists, elbows, knees and so on), as 3D joint positions, or as a full 3D body mesh fitted to a parametric model such as SMPL (2015). It builds on keypoint detectionkeypoint detection, n.: Finding specific, recognizable points in an image, such as the corners of a building, a person's elbows and knees, or the tip of a nose.: top-down systems detect each person and then predict heatmaps for their joints (HRNet, 2019; ViTPose, 2022), while bottom-up systems find all joints and link them into skeletons, as OpenPose (2017) did with 'part affinity fields'. DeepPose (2014) was the first to regress joint positions with a deep network. Hands, faces and animals are handled the same way. Benchmarks include COCOCOCO, prop. n.: A widely used collection of everyday photos in which people outlined and named the objects, described each picture in sentences and marked body joints; it is used to train and compare vision systems. keypoints, MPII and, for 3D, Human3.6M. Applications include fitness and physiotherapy apps, sports analysis, motion capture for animation, sign-language research, ergonomics, and teaching robots by demonstration.
Pose estimation shows the runner's knee collapsing inward on landing, which the coach had missed.
- 2
Objects and cameras
Estimating the six-degree-of-freedomdegrees of freedom, n.: The number of independent ways something can move. A robot arm with six motorized joints has six; a free-floating object can slide three ways and turn three ways. pose (3D rotation plus 3D translation) of a rigid object or a camera relative to a reference frame. Classical methods match 2D image points to known 3D points and solve the Perspective-n-Point problem, usually inside the RANSAC algorithm for robustness to bad matches; learned methods regress the pose directly or predict correspondences. Object pose estimation is central to robotic grasping and augmented reality; camera pose estimation underlies structure from motionstructure from motion, n.: Recovering both the 3D shape of a scene and the spot from which each photo was taken, using nothing but a collection of overlapping photographs., visual odometryodometry, n.: Estimating how far and in which direction a robot or vehicle has moved by adding up its own motion, such as wheel turns or how camera images shift, like counting your steps. and simultaneous localization and mappingsimultaneous localization and mapping, n.: How a robot, drone or headset builds a map of an unfamiliar place while at the same time working out where it is on that map, using its cameras or laser scanners..
Origin
Part-based models of the human figure go back to Martin Fischler and Robert Elschlager's 'pictorial structures' (1973); recovering camera pose from point correspondences is older still, rooted in photogrammetry. Deep human pose estimation began with Alexander Toshev and Christian Szegedy's DeepPose (2014), and Zhe Cao and colleagues' OpenPose (2017) made real-time multi-person estimation widely available.
Usage note
In robotics, 'pose' means position plus orientation; in human pose estimation it means body configuration. Body-movement data can reveal identity (through gait), health and emotional state, which raises privacy questions.
Related
- keypoint detectionkeypoint detection, n.: Finding specific, recognizable points in an image, such as the corners of a building, a person's elbows and knees, or the tip of a nose.
- action recognitionaction recognition, n.: Teaching computers to recognize what people are doing in a video, such as running, waving, cooking or falling, rather than just which objects appear in it.
- object trackingobject tracking, n.: Following objects from frame to frame in a video and keeping track of which is which, so a system knows that the car it sees now is the same car it saw a second ago.
- structure from motionstructure from motion, n.: Recovering both the 3D shape of a scene and the spot from which each photo was taken, using nothing but a collection of overlapping photographs.
- simultaneous localization and mappingsimultaneous localization and mapping, n.: How a robot, drone or headset builds a map of an unfamiliar place while at the same time working out where it is on that map, using its cameras or laser scanners.
- degrees of freedomdegrees of freedom, n.: The number of independent ways something can move. A robot arm with six motorized joints has six; a free-floating object can slide three ways and turn three ways.
Sources
- Toshev, A. & Szegedy, C. (2014). DeepPose: Human Pose Estimation via Deep Neural Networks. CVPR 2014.
- Cao, Z., Simon, T., Wei, S.-E. & Sheikh, Y. (2017). Realtime Multi-Person 2D Pose Estimation Using Part Affinity Fields. CVPR 2017.
Mentioned in
ControlNetControlNet, prop. n.: An add-on for image generators that makes them follow a guide image — a sketch, a stick-figure pose or a depth map — so the layout comes out as intended., graspinggrasping, n.: How a robot picks up and holds an object securely with a gripper or hand: deciding where to put its fingers or suction cup and how hard to squeeze., keypoint detectionkeypoint detection, n.: Finding specific, recognizable points in an image, such as the corners of a building, a person's elbows and knees, or the tip of a nose., YOLOYOLO, prop. n.: A family of fast object-detection models that look at an image once and immediately output boxes and labels for everything they recognize, quick enough for live video., action recognitionaction recognition, n.: Teaching computers to recognize what people are doing in a video, such as running, waving, cooking or falling, rather than just which objects appear in it., structure from motionstructure from motion, n.: Recovering both the 3D shape of a scene and the spot from which each photo was taken, using nothing but a collection of overlapping photographs.