sensor fusion
In plain wordsCombining readings from several different sensors, such as cameras, radar, lidar, GPS and motion sensors, to get a more accurate and reliable picture than any one sensor gives alone.
The merging of measurements from several sensors, or from one sensor over time, into a single estimate of the state of a system or its surroundings, with the aim of beating every individual source on precision, coverage or robustness to failure. Sensors are fused because their strengths complement each other: cameras give texture and semantics, lidarlidar, n.: A sensor that measures distances by firing laser pulses and timing their reflections, building a 3D picture of its surroundings out of millions of points; widely used on robots and self-driving cars. precise geometry, radar range and velocity in bad weather, inertial sensors fast motion, and GPS absolute position. Statistical fusion weights each measurement by its uncertainty, most often with a Kalman filterKalman filter, n.: A method that keeps a running best guess of something you cannot measure exactly, such as a vehicle's position, by blending predictions from a motion model with noisy sensor readings. or its nonlinear variants — GPS/inertial navigation is the classic case — or with a particle filterparticle filter, n.: A way to track something uncertain, such as a robot's location, by keeping thousands of random guesses, scoring each against what the sensors see, and copying the best ones. or factor-graph optimization. In perception for autonomous vehiclesautonomous vehicle, n.: A vehicle, usually a car, truck or shuttle, that can drive itself, sensing the road, deciding what to do and steering, braking and accelerating with no human driving., fusion is described by stage: early fusion combines raw data before detection, mid-level fusion combines learned features, increasingly in a shared bird's-eye-view representation built by transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. networks, and late fusion reconciles independent detections. Fusion requires careful calibration and time synchronization between sensors and adds redundancy when one sensor fails or is fooled.
Sensor fusion kept the car tracking the cyclist after glare blinded the front camera.
Origin
Developed from the 1970s for military tracking, surveillance and navigation, where the US Joint Directors of Laboratories (JDL) data-fusion group's model of fusion levels, issued in the late 1980s, became a standard reference; the Kalman filter (1960) supplied the core estimation method.
Usage note
In deep learning, 'fusion' also means combining modalities inside one network; the robotics sense emphasizes estimating physical state under uncertainty.
Related
- Kalman filterKalman filter, n.: A method that keeps a running best guess of something you cannot measure exactly, such as a vehicle's position, by blending predictions from a motion model with noisy sensor readings.
- particle filterparticle filter, n.: A way to track something uncertain, such as a robot's location, by keeping thousands of random guesses, scoring each against what the sensors see, and copying the best ones.
- lidarlidar, n.: A sensor that measures distances by firing laser pulses and timing their reflections, building a 3D picture of its surroundings out of millions of points; widely used on robots and self-driving cars.
- odometryodometry, n.: Estimating how far and in which direction a robot or vehicle has moved by adding up its own motion, such as wheel turns or how camera images shift, like counting your steps.
- simultaneous localization and mappingsimultaneous localization and mapping, n.: How a robot, drone or headset builds a map of an unfamiliar place while at the same time working out where it is on that map, using its cameras or laser scanners.
- autonomous vehicleautonomous vehicle, n.: A vehicle, usually a car, truck or shuttle, that can drive itself, sensing the road, deciding what to do and steering, braking and accelerating with no human driving.
- multimodal AImultimodal AI, n.: AI that can take in or produce more than one kind of information, such as text, images, sound and video, so you can show it a photo and ask questions about it.
Sources
- Durrant-Whyte, H. & Henderson, T. C. (2016). Multisensor data fusion. In Springer Handbook of Robotics, 2nd ed. Springer.
Mentioned in
autonomous vehicleautonomous vehicle, n.: A vehicle, usually a car, truck or shuttle, that can drive itself, sensing the road, deciding what to do and steering, braking and accelerating with no human driving., advanced driver-assistance systemadvanced driver-assistance system, n.: Car features that help the driver but do not replace them, such as automatic emergency braking, lane-keeping help, adaptive cruise control and blind-spot warnings., calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., Kalman filterKalman filter, n.: A method that keeps a running best guess of something you cannot measure exactly, such as a vehicle's position, by blending predictions from a motion model with noisy sensor readings., lidarlidar, n.: A sensor that measures distances by firing laser pulses and timing their reflections, building a 3D picture of its surroundings out of millions of points; widely used on robots and self-driving cars., perceptionperception, n.: How an AI system or robot turns raw sensor signals, such as camera pixels, microphone sound or laser scans, into an understanding of what is around it: objects, people, words and distances., simultaneous localization and mappingsimultaneous localization and mapping, n.: How a robot, drone or headset builds a map of an unfamiliar place while at the same time working out where it is on that map, using its cameras or laser scanners., WaymoWaymo, prop. n.: A company owned by Google's parent, Alphabet, that runs fully driverless robotaxi services in several American cities., occupancy gridoccupancy grid, n.: A robot's map of the world drawn as a grid of small squares or cubes, each marked with the chance that something solid is there, so the robot knows where it can safely go., odometryodometry, n.: Estimating how far and in which direction a robot or vehicle has moved by adding up its own motion, such as wheel turns or how camera images shift, like counting your steps., particle filterparticle filter, n.: A way to track something uncertain, such as a robot's location, by keeping thousands of random guesses, scoring each against what the sensors see, and copying the best ones., proprioceptionproprioception, n.: A robot's sense of its own body: where its joints are, how fast they are moving and how hard its motors are pushing, measured by internal sensors rather than cameras., tactile sensingtactile sensing, n.: Giving robots a sense of touch, with sensors in fingertips or skin that feel pressure, texture, vibration and slipping, so they can handle objects gently and securely.