loss landscape
In plain wordsA way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys.
The lossloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down. viewed as a surface over the space of a model's parameters, so that training is a descent across it. For neural networks the space has millions to trillions of dimensions and the surface is highly non-convex, with many saddle pointssaddle point, n.: A spot on a surface that is a low point in one direction but a high point in another, like the middle of a horse's saddle or a mountain pass., wide plateaus, narrow ravines and large connected regions of near-equal low loss. Since it cannot be seen directly, it is studied through one- and two-dimensional slices (with 'filter normalization' to make slices comparable across networks), interpolation between weightweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. vectors, and the HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass. spectrum, whose largest eigenvalues measure sharpness. Findings that shaped practice include: most critical points in high dimensions are saddles rather than bad local minima; skip connectionsresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable. and normalization make landscapes smoother; separately trained solutions are often joined by low-loss paths (mode connectivitymode connectivity, n.: The discovery that two separately trained versions of a network, though their settings look very different, can usually be joined by a path along which the model keeps working well.); and curvature evolves during training (edge of stabilityedge of stability, n.: A surprising pattern in how neural networks train: the error landscape keeps getting steeper until it is just steep enough to make training wobble, then stays balanced at that edge while the error still falls.). The geometry around a solution is linked, contentiously, to generalization (see sharp and flat minimasharp and flat minima, n.: Two kinds of low points a trained model can settle into: narrow pits where small changes spoil performance, and broad basins where they barely matter. Broad basins are thought to generalize better, though this is debated.).
Origin
The imagery comes from the 'adaptive landscape' that Sewall Wright introduced in evolutionary biology (1932), now usually called a fitness landscape, and from energy landscapes in physics, which shaped 1980s work on Hopfield networks. 'Error surface' was standard in the 1980s backpropagation literature; 'loss landscape' spread in the 2010s, helped by Hao Li and colleagues' widely reproduced visualizations (arXiv 2017; NeurIPS 2018).
Usage note
Low-dimensional pictures can mislead: a slice may look convex or chaotic depending on the directions chosen, and rescaling parameters can change the apparent landscape without changing the function the network computes.
Related
- sharp and flat minimasharp and flat minima, n.: Two kinds of low points a trained model can settle into: narrow pits where small changes spoil performance, and broad basins where they barely matter. Broad basins are thought to generalize better, though this is debated.
- mode connectivitymode connectivity, n.: The discovery that two separately trained versions of a network, though their settings look very different, can usually be joined by a path along which the model keeps working well.
- saddle pointsaddle point, n.: A spot on a surface that is a low point in one direction but a high point in another, like the middle of a horse's saddle or a mountain pass.
- edge of stabilityedge of stability, n.: A surprising pattern in how neural networks train: the error landscape keeps getting steeper until it is just steep enough to make training wobble, then stays balanced at that edge while the error still falls.
- HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass.
- gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.
- singular learning theorysingular learning theory, n.: A mathematical theory of learning for models like neural networks, whose many redundant settings break standard statistics; it explains how such models generalize and how their learning unfolds.
Sources
- Li, H., Xu, Z., Taylor, G., Studer, C. & Goldstein, T. (2018). Visualizing the Loss Landscape of Neural Nets. NeurIPS 2018. arxiv.org/abs/1712.09913
- Dauphin, Y. et al. (2014). Identifying and attacking the saddle point problem in high-dimensional non-convex optimization. NeurIPS 2014.
Mentioned in
gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down., loss functionloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down., batch normalizationbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably., convex optimizationconvex optimization, n.: Finding the lowest point of a bowl-shaped landscape. Because a bowl has only one bottom, such problems can be solved reliably — unlike training a neural network, whose landscape is bumpy., eigenvectoreigenvector, n.: A special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies., gradient clippinggradient clipping, n.: A safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model., local minimumlocal minimum, n.: A spot lower than everything immediately around it, like the bottom of a small valley, even though a deeper valley may exist somewhere else., model mergingmodel merging, n.: Combining two or more trained AI models into one by blending their internal numbers directly, so the result shares their skills without any further training., momentummomentum, n.: A training trick that lets each update keep part of the previous update's direction, like a ball rolling downhill that builds up speed and coasts over small bumps., developmental interpretabilitydevelopmental interpretability, n.: The study of how a neural network's internal structure forms during training — when and how new abilities and mechanisms appear — rather than only examining the finished model., edge of stabilityedge of stability, n.: A surprising pattern in how neural networks train: the error landscape keeps getting steeper until it is just steep enough to make training wobble, then stays balanced at that edge while the error still falls., local learning coefficientlocal learning coefficient, n.: A number measuring how complex a trained neural network effectively is around its current settings — usually far less than its raw parameter count suggests — used to spot changes as the network learns., loss of plasticityloss of plasticity, n.: The gradual loss of a neural network's ability to learn new things after long training on changing tasks, as parts of the network go inactive or become stuck., mode connectivitymode connectivity, n.: The discovery that two separately trained versions of a network, though their settings look very different, can usually be joined by a path along which the model keeps working well., model soupmodel soup, n.: A single model made by averaging the internal numbers of several copies of a model that were fine-tuned in slightly different ways, often doing better than any one of them., overparameterizationoverparameterization, n.: Giving a model far more adjustable settings than it would seem to need — often more than it has training examples. Classic theory predicted this would backfire; in deep learning it usually helps., saddle pointsaddle point, n.: A spot on a surface that is a low point in one direction but a high point in another, like the middle of a horse's saddle or a mountain pass., second-order optimizationsecond-order optimization, n.: Training methods that use not only the slope of the error but also how that slope is changing — its curvature — to choose better-sized steps. They can need far fewer steps, but each costs more., sharp and flat minimasharp and flat minima, n.: Two kinds of low points a trained model can settle into: narrow pits where small changes spoil performance, and broad basins where they barely matter. Broad basins are thought to generalize better, though this is debated., sharpness-aware minimizationsharpness-aware minimization, n.: A training method that looks for settings where the error stays low even if the settings are nudged a little, rather than settings that are merely low at one precise point., singular learning theorysingular learning theory, n.: A mathematical theory of learning for models like neural networks, whose many redundant settings break standard statistics; it explains how such models generalize and how their learning unfolds.