sharp and flat minima
In plain wordsTwo kinds of low points a trained model can settle into: narrow pits where small changes spoil performance, and broad basins where they barely matter. Broad basins are thought to generalize better, though this is debated.
A distinction between minima of a loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys. by how quickly the loss rises around them: at a sharp minimum it climbs steeply in some directions (large HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass. eigenvalues), while a flat or wide minimum sits in a broad region of low loss. The flat-minima hypothesis holds that flat solutions generalize better, because they can be specified with fewer bits of precision (a minimum description lengthminimum description length, n.: The principle that the best explanation of some data is the one that lets you write the data down most briefly, counting both the rule and its exceptions. Good theories compress. argument) and are robust to small differences between training and test loss surfaces. Supporting evidence includes the finding that large-batch training reaches sharper minima and generalizes worse, and the success of flatness-seeking methods such as sharpness-aware minimizationsharpness-aware minimization, n.: A training method that looks for settings where the error stays low even if the settings are nudged a little, rather than settings that are merely low at one precise point. and weight averaging. Critics showed that in ReLUReLU, abbr.: The most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective. networks, rescaling weights layer by layer can make any minimum arbitrarily sharp without changing the function computed, so naive sharpness cannot by itself explain generalization; reparametrization-invariant measures and PAC-Bayes bounds are proposed responses, and the question remains open.
Origin
Sepp Hochreiter and Jürgen Schmidhuber proposed searching for flat minima in a 1994 NeurIPS paper and 'Flat Minima' (Neural Computation, 1997). Nitish Shirish Keskar and colleagues revived the idea in 2016 to explain the large-batch generalization gap, and Laurent Dinh, Razvan Pascanu, Samy Bengio and Yoshua Bengio's 'Sharp Minima Can Generalize for Deep Nets' (2017) challenged it.
Related
- loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys.
- sharpness-aware minimizationsharpness-aware minimization, n.: A training method that looks for settings where the error stays low even if the settings are nudged a little, rather than settings that are merely low at one precise point.
- batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step.
- generalization gapgeneralization gap, n.: The difference between how well a model does on the examples it trained on and how well it does on new ones. A big gap means it memorized rather than learned.
- minimum description lengthminimum description length, n.: The principle that the best explanation of some data is the one that lets you write the data down most briefly, counting both the rule and its exceptions. Good theories compress.
- HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass.
- edge of stabilityedge of stability, n.: A surprising pattern in how neural networks train: the error landscape keeps getting steeper until it is just steep enough to make training wobble, then stays balanced at that edge while the error still falls.
Sources
- Hochreiter, S. & Schmidhuber, J. (1997). Flat Minima. Neural Computation 9(1), 1–42.
- Keskar, N. S. et al. (2017). On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. ICLR 2017. arxiv.org/abs/1609.04836
- Dinh, L., Pascanu, R., Bengio, S. & Bengio, Y. (2017). Sharp Minima Can Generalize For Deep Nets. ICML 2017. arxiv.org/abs/1703.04933
Mentioned in
batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step., generalizationgeneralization, n.: How well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers., generalization gapgeneralization gap, n.: The difference between how well a model does on the examples it trained on and how well it does on new ones. A big gap means it memorized rather than learned., local minimumlocal minimum, n.: A spot lower than everything immediately around it, like the bottom of a small valley, even though a deeper valley may exist somewhere else., loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys., stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset., edge of stabilityedge of stability, n.: A surprising pattern in how neural networks train: the error landscape keeps getting steeper until it is just steep enough to make training wobble, then stays balanced at that edge while the error still falls., HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass., implicit regularizationimplicit regularization, n.: The hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions., sharpness-aware minimizationsharpness-aware minimization, n.: A training method that looks for settings where the error stays low even if the settings are nudged a little, rather than settings that are merely low at one precise point.