invariance
In plain wordsA property of a model whose answer does not change when the input is altered in an irrelevant way, such as a photo classifier that still says 'cat' when the cat moves or the picture is flipped.
The property of a function f that is unchanged by a group of input transformations: f(g·x) = f(x) for every g in the group. Classifiers usually should be invariant to nuisance transformations (translation, small rotations, lighting, the order of elements in a set) while staying sensitive to label-relevant changes. Invariance can be built into an architecture by pooling equivariantequivariance, n.: A property of a model whose output changes in step with its input: move the cat in a photo, and the network's internal 'cat' signal moves the same way. features (global poolingpooling, n.: A step in an image network that shrinks its internal picture by summarizing each small patch with one number, usually the largest or the average, keeping what matters and discarding exact positions. after convolutions gives approximate translation invariance; sum or mean readouts give permutation invariance, as in Deep Sets and graph-level GNNgraph neural network, n.: A neural network for data shaped like a web of connections, such as molecules, social networks or road maps, in which each point repeatedly gathers information from its neighbors. outputs), or encouraged during training through data augmentationdata augmentation, n.: Creating extra training examples by changing existing ones in ways that keep their meaning — a flipped, cropped or recolored photo of a cat still shows a cat — so the model learns more robustly. or contrastive learningcontrastive learning, n.: A way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things. between transformed views. Too much invariance discards information: full rotation invariance confuses a '6' with a '9'. Capsule-networkcapsule network, n.: A network design championed by Geoffrey Hinton in which groups of neurons, called capsules, each describe a part of an object and its position, so the network can check whether the parts fit together. proponents argued that CNNs pursued invariance where equivariance was needed.
Random crops during training make the classifier more invariant to where the object appears.
Origin
Invariance under transformation groups is an old idea in mathematics, organizing Felix Klein's Erlangen program (1872), and in the study of perception. It shaped pattern-recognition research from the 1950s and Kunihiko Fukushima's neocognitron (1980), designed for recognition 'unaffected by shift in position'; Manzil Zaheer and colleagues' Deep Sets (2017) characterized permutation-invariant networks.
Related
- equivarianceequivariance, n.: A property of a model whose output changes in step with its input: move the cat in a photo, and the network's internal 'cat' signal moves the same way.
- poolingpooling, n.: A step in an image network that shrinks its internal picture by summarizing each small patch with one number, usually the largest or the average, keeping what matters and discarding exact positions.
- data augmentationdata augmentation, n.: Creating extra training examples by changing existing ones in ways that keep their meaning — a flipped, cropped or recolored photo of a cat still shows a cat — so the model learns more robustly.
- geometric deep learninggeometric deep learning, n.: An approach to designing neural networks around the shapes and symmetries of the data, whether grids, graphs, surfaces or 3-D structures, so the network respects how that data can be moved or rearranged.
- convolutional neural networkconvolutional neural network, n.: A kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects.
- inductive biasinductive bias, n.: The assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases.
- equivarianceequivariance, n.: A property of a model whose output changes in step with its input: move the cat in a photo, and the network's internal 'cat' signal moves the same way.
Mentioned in
poolingpooling, n.: A step in an image network that shrinks its internal picture by summarizing each small patch with one number, usually the largest or the average, keeping what matters and discarding exact positions., capsule networkcapsule network, n.: A network design championed by Geoffrey Hinton in which groups of neurons, called capsules, each describe a part of an object and its position, so the network can check whether the parts fit together., equivarianceequivariance, n.: A property of a model whose output changes in step with its input: move the cat in a photo, and the network's internal 'cat' signal moves the same way., geometric deep learninggeometric deep learning, n.: An approach to designing neural networks around the shapes and symmetries of the data, whether grids, graphs, surfaces or 3-D structures, so the network respects how that data can be moved or rearranged.