implicit neural representation
In plain wordsA way of storing an image, shape or scene as a small neural network that, given any coordinate, returns what is there, such as a color or density, instead of as a grid of pixels.
A way of encoding a signal, such as an image, a 3-D shape or a sound, in the weights of a neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain., typically a multilayer perceptronmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model., that takes a coordinate (pixel position, 3-D point, time, viewing direction) and returns the signal's value there (color, occupancy, signed distance, density, sound pressure). Because that function is continuous, the signal has no fixed resolution and can be queried anywhere and differentiated with respect to its inputs. Plain ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective. networks are biased towards smooth functions, so implicit neural representations use Fourier-feature input encodings, sine activations (SIREN) or multiresolution hash-grid encodings to capture fine detail. Applications include 3-D shape modeling, neural radiance fieldsneural radiance field, n.: A way of capturing a real scene from a set of photos as a small neural network that can then show the scene from any new viewpoint, as if moving a camera through it. for view synthesis, compression, super-resolution and solving differential equations; hypernetworkshypernetwork, n.: A neural network whose job is to produce the settings, or weights, of another neural network, so that one model can generate many specialized models on demand. and meta-learning can produce such representations for many signals at once.
Each MRI volume is stored as an implicit neural representation and resampled at any slice thickness.
Origin
Coordinate-based networks for 3-D shape emerged in 2019 with Occupancy Networks (Lars Mescheder and colleagues) and DeepSDF (Jeong Joon Park and colleagues). Ben Mildenhall and colleagues' NeRF (2020) and Vincent Sitzmann and colleagues' 'Implicit Neural Representations with Periodic Activation Functions' (June 2020), which introduced SIREN and helped fix the name, made the approach widespread.
Related
- neural radiance fieldneural radiance field, n.: A way of capturing a real scene from a set of photos as a small neural network that can then show the scene from any new viewpoint, as if moving a camera through it.
- multilayer perceptronmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model.
- hypernetworkhypernetwork, n.: A neural network whose job is to produce the settings, or weights, of another neural network, so that one model can generate many specialized models on demand.
- Gaussian splattingGaussian splatting, n.: A way of turning photos of a real place into a 3D scene made of millions of tiny, soft, colored blobs that can be viewed from any angle in real time, even on a phone or headset.
- super-resolutionsuper-resolution, n.: Making a small or blurry image larger and sharper by having AI fill in plausible fine detail, rather than just stretching the pixels.
- physics-informed neural networkphysics-informed neural network, n.: A neural network that is penalized during training whenever its output breaks a known physical law, such as an equation of fluid flow, so it can model a system well even from few measurements.
Sources
- Sitzmann, V., Martel, J. N. P., Bergman, A. W., Lindell, D. B. & Wetzstein, G. (2020). Implicit Neural Representations with Periodic Activation Functions. arXiv:2006.09661; NeurIPS 2020. arxiv.org/abs/2006.09661
Mentioned in
physics-informed neural networkphysics-informed neural network, n.: A neural network that is penalized during training whenever its output breaks a known physical law, such as an equation of fluid flow, so it can model a system well even from few measurements.