Skip to content
Last Dictionary
Plane
Vision, Speech & PerceptionGenerative MediaDeep Learning

neural radiance field

noun/nurf/Also: NeRF, radiance field, Instant-NGP, novel view synthesis

In plain wordsA way of capturing a real scene from a set of photos as a small neural network that can then show the scene from any new viewpoint, as if moving a camera through it.

  1. A scene representation in which a function, usually a small neural network, assigns every 3D point a volume density and, depending on the direction of view, an emitted color; its main use is novel view synthesis, producing realistic images from viewpoints absent from the input photos. To form an image, each pixel's ray is sampled at many points whose colors are blended front to back, each weighted by its density and by the share of light surviving to it. Since that blending is differentiable, the network is trained simply by making its renderings match a few dozen to a few hundred photos whose camera poses are known (usually from structure from motionstructure from motion, n.: Recovering both the 3D shape of a scene and the spot from which each photo was taken, using nothing but a collection of overlapping photographs.); a Fourier-feature positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model. of the inputs lets a small multilayer perceptronmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model. represent fine detail. The original method took hours to days per scene; multiresolution hash grids (Instant-NGP, 2022) cut training to minutes, and anti-aliased, unbounded variants (Mip-NeRF 360, Zip-NeRF) raised quality. From 2023, Gaussian splattingGaussian splatting, n.: A way of turning photos of a real place into a 3D scene made of millions of tiny, soft, colored blobs that can be viewed from any angle in real time, even on a phone or headset. displaced NeRFs in many applications because it renders in real time.

    We captured the lobby with a phone and trained a NeRF overnight for the virtual walkthrough.

C(r)=∫tntfT(t) σ(r(t)) c(r(t),d) dt,T(t)=exp⁡ ⁣(−∫tntσ(r(s)) ds)C(\mathbf{r}) = \int_{t_n}^{t_f} T(t)\,\sigma(\mathbf{r}(t))\,\mathbf{c}(\mathbf{r}(t),\mathbf{d})\,dt, \quad T(t) = \exp\!\left(-\int_{t_n}^{t}\sigma(\mathbf{r}(s))\,ds\right)

Origin

Introduced by Ben Mildenhall, Pratul Srinivasan, Matthew Tancik, Jonathan Barron, Ravi Ramamoorthi and Ren Ng (UC Berkeley, Google Research, UC San Diego) in 'NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis' (arXiv March 2020; ECCV 2020), which set off a large wave of follow-up research.

Usage note

In the formula, C(r) is the color of camera ray r, σ the density, c the color seen from direction d and T the transmittance. A NeRF is fitted to one scene; it is a representation and rendering method rather than a generative model, though generative systems can output NeRFs.

Related

See also
  • Gaussian splattingGaussian splatting, n.: A way of turning photos of a real place into a 3D scene made of millions of tiny, soft, colored blobs that can be viewed from any angle in real time, even on a phone or headset.
  • 3D reconstruction3D reconstruction, n.: Building a 3D model of an object or place, its shape and often its colors, from photos, video or depth-sensor scans.
  • structure from motionstructure from motion, n.: Recovering both the 3D shape of a scene and the spot from which each photo was taken, using nothing but a collection of overlapping photographs.
  • positional encodingpositional encoding, n.: Extra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model.
  • multilayer perceptronmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model.
  • depth estimationdepth estimation, n.: Working out how far away each point in a scene is from the camera, producing a 'depth map' that turns a flat picture into a sense of near and far.
Contrast with
  • Gaussian splattingGaussian splatting, n.: A way of turning photos of a real place into a 3D scene made of millions of tiny, soft, colored blobs that can be viewed from any angle in real time, even on a phone or headset.

Sources

  1. Mildenhall, B., Srinivasan, P. P., Tancik, M., Barron, J. T., Ramamoorthi, R. & Ng, R. (2020). NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis. ECCV 2020. arxiv.org/abs/2003.08934
  2. Müller, T., Evans, A., Schied, C. & Keller, A. (2022). Instant Neural Graphics Primitives with a Multiresolution Hash Encoding. ACM Transactions on Graphics 41(4).

Mentioned in

3D reconstruction3D reconstruction, n.: Building a 3D model of an object or place, its shape and often its colors, from photos, video or depth-sensor scans., Gaussian splattingGaussian splatting, n.: A way of turning photos of a real place into a 3D scene made of millions of tiny, soft, colored blobs that can be viewed from any angle in real time, even on a phone or headset., simultaneous localization and mappingsimultaneous localization and mapping, n.: How a robot, drone or headset builds a map of an unfamiliar place while at the same time working out where it is on that map, using its cameras or laser scanners., implicit neural representationimplicit neural representation, n.: A way of storing an image, shape or scene as a small neural network that, given any coordinate, returns what is there, such as a color or density, instead of as a grid of pixels., nerfednerfed, adj.: Gaming slang for something made weaker by an update; applied to AI models that users believe got worse after a provider changed them, often without saying so., score distillation samplingscore distillation sampling, n.: A trick for creating 3D objects from a text description by repeatedly asking a 2D image generator whether pictures of the object, seen from many angles, look right — and adjusting the object until they do., spatial intelligencespatial intelligence, n.: The ability to understand, imagine and act in three-dimensional space and the physical world — effortless for people, and treated by many AI researchers as the next big goal after language., structure from motionstructure from motion, n.: Recovering both the 3D shape of a scene and the spot from which each photo was taken, using nothing but a collection of overlapping photographs.