Gaussian splatting
In plain wordsA way of turning photos of a real place into a 3D scene made of millions of tiny, soft, colored blobs that can be viewed from any angle in real time, even on a phone or headset.
A scene representation and rendering method that models a 3D scene as a large set, often millions, of anisotropic 3D Gaussians, each with a position, a covariance (stored as a scale and a rotation), an opacity and a view-dependent color encoded with spherical harmonics. To render, the Gaussians are projected ('splatted') onto the image plane as 2D ellipses, sorted by depth within screen tiles and alpha-blended front to back by a fast, differentiable rasterizer. Starting from the sparse point cloudpoint cloud, n.: A 3D picture made of many dots floating in space, each marking a spot on a surface, as captured by a laser scanner or worked out from photos. produced by structure from motionstructure from motion, n.: Recovering both the 3D shape of a scene and the spot from which each photo was taken, using nothing but a collection of overlapping photographs., their parameters are optimized by gradient descent to reproduce the input photos, while adaptive density control clones, splits and prunes Gaussians where detail is missing or wasted. Compared with neural radiance fieldsneural radiance field, n.: A way of capturing a real scene from a set of photos as a small neural network that can then show the scene from any new viewpoint, as if moving a camera through it., it trains in minutes and renders at 30 frames per second or more at 1080p resolution, at the cost of more memory. Extensions cover dynamic scenes (4D Gaussians), animatable avatars, compression, SLAMsimultaneous localization and mapping, n.: How a robot, drone or headset builds a map of an unfamiliar place while at the same time working out where it is on that map, using its cameras or laser scanners. and feed-forward prediction of Gaussians from a few images, and splats have been adopted in mainstream 3D tools and web viewers.
The real-estate scan exports as a 300 MB Gaussian splat that streams smoothly in a browser.
Origin
Introduced by Bernhard Kerbl, Georgios Kopanas, Thomas Leimkühler and George Drettakis (Inria and Université Côte d'Azur; Max Planck Institute for Informatics) in '3D Gaussian Splatting for Real-Time Radiance Field Rendering' (SIGGRAPH 2023; arXiv August 2023). 'Splatting' comes from Lee Westover's volume-rendering technique of about 1990, extended by elliptical weighted average splatting (2001).
Usage note
Like NeRF, Gaussian splatting reconstructs a captured scene and is not in itself generative; reflections, thin structures and areas seen from few viewpoints remain common failure points. 'Splat' is used informally for a captured scene.
Related
- neural radiance fieldneural radiance field, n.: A way of capturing a real scene from a set of photos as a small neural network that can then show the scene from any new viewpoint, as if moving a camera through it.
- 3D reconstruction3D reconstruction, n.: Building a 3D model of an object or place, its shape and often its colors, from photos, video or depth-sensor scans.
- structure from motionstructure from motion, n.: Recovering both the 3D shape of a scene and the spot from which each photo was taken, using nothing but a collection of overlapping photographs.
- point cloudpoint cloud, n.: A 3D picture made of many dots floating in space, each marking a spot on a surface, as captured by a laser scanner or worked out from photos.
- digital twindigital twin, n.: A detailed virtual copy of a real machine, factory, robot or city, kept in sync with live sensor data and used to monitor it, test changes safely and predict problems.
- neural radiance fieldneural radiance field, n.: A way of capturing a real scene from a set of photos as a small neural network that can then show the scene from any new viewpoint, as if moving a camera through it.
Sources
- Kerbl, B., Kopanas, G., Leimkühler, T. & Drettakis, G. (2023). 3D Gaussian Splatting for Real-Time Radiance Field Rendering. ACM Transactions on Graphics 42(4) (SIGGRAPH 2023). arxiv.org/abs/2308.04079
- Zwicker, M., Pfister, H., van Baar, J. & Gross, M. (2001). EWA Volume Splatting. Proceedings of IEEE Visualization 2001.
Mentioned in
3D reconstruction3D reconstruction, n.: Building a 3D model of an object or place, its shape and often its colors, from photos, video or depth-sensor scans., neural radiance fieldneural radiance field, n.: A way of capturing a real scene from a set of photos as a small neural network that can then show the scene from any new viewpoint, as if moving a camera through it., point cloudpoint cloud, n.: A 3D picture made of many dots floating in space, each marking a spot on a surface, as captured by a laser scanner or worked out from photos., sim-to-real transfersim-to-real transfer, n.: Training a robot's AI in a computer simulation, where practice is fast, cheap and safe, and then making it work on the real robot despite the differences between simulation and reality., simultaneous localization and mappingsimultaneous localization and mapping, n.: How a robot, drone or headset builds a map of an unfamiliar place while at the same time working out where it is on that map, using its cameras or laser scanners., implicit neural representationimplicit neural representation, n.: A way of storing an image, shape or scene as a small neural network that, given any coordinate, returns what is there, such as a color or density, instead of as a grid of pixels., score distillation samplingscore distillation sampling, n.: A trick for creating 3D objects from a text description by repeatedly asking a 2D image generator whether pictures of the object, seen from many angles, look right — and adjusting the object until they do., spatial intelligencespatial intelligence, n.: The ability to understand, imagine and act in three-dimensional space and the physical world — effortless for people, and treated by many AI researchers as the next big goal after language., structure from motionstructure from motion, n.: Recovering both the 3D shape of a scene and the spot from which each photo was taken, using nothing but a collection of overlapping photographs.