Skip to content
Last Dictionary
Plane
Interpretability & SteeringMathematics & StatisticsDeep Learning

singular learning theory

nounAlso: SLT, learning coefficient, real log canonical threshold, RLCT

In plain wordsA mathematical theory of learning for models like neural networks, whose many redundant settings break standard statistics; it explains how such models generalize and how their learning unfolds.

  1. A Bayesian theory of statistical learning for singular models — models in which the map from parameters to probability distributions is not one-to-one and the Fisher information matrix is degenerate, as in neural networks, mixture models and hidden Markov models. Classical results such as the Bayesian information criterion assume regular models and fail for them. Using resolution of singularities from algebraic geometry, Sumio Watanabe showed that the asymptotic Bayesian free energy and generalization error are governed not by the parameter count d but by the real log canonical threshold, or learning coefficient λ, which satisfies λ ≤ d/2 and is smaller for more degenerate solutions. Bayesian learning therefore favors degenerate, effectively simpler solutions — a form of Occam's razorOccam's razor, n.: A rule of thumb for choosing between explanations: if two fit the evidence equally well, bet on the one that assumes less. In AI it motivates preferring simpler models, which are less likely to fit noise. — and can pass through phase transitions as data accumulate. Derived tools include the WAIC and WBIC information criteria. In AI research SLT underpins developmental interpretabilitydevelopmental interpretability, n.: The study of how a neural network's internal structure forms during training — when and how new abilities and mechanisms appear — rather than only examining the finished model. and the local learning coefficientlocal learning coefficient, n.: A number measuring how complex a trained neural network effectively is around its current settings — usually far less than its raw parameter count suggests — used to spot changes as the network learns..

    Singular learning theory says the parameter count is the wrong measure of this network's complexity.

Fn=nLn(w0)+λlog⁡n−(m−1)log⁡log⁡n+Op(1)F_n = n L_n(w_0) + \lambda \log n - (m - 1) \log \log n + O_p(1)

Origin

Developed by Sumio Watanabe from the late 1990s, with key papers around 1999–2001, and set out in his monographs 'Algebraic Geometry and Statistical Learning Theory' (Cambridge University Press, 2009) and 'Mathematical Theory of Bayesian Statistics' (2018). Daniel Murfet and colleagues brought it into deep-learning interpretability and AI safety research in the early 2020s.

Usage note

In the formula, F_n is the Bayesian free energy (negative log marginal likelihood), L_n the empirical loss at the optimal parameter w₀, λ the learning coefficient and m its multiplicity. 'Singular' refers to the geometry of the parameter space, not to singular matrices in general.

Related

See also
  • local learning coefficientlocal learning coefficient, n.: A number measuring how complex a trained neural network effectively is around its current settings — usually far less than its raw parameter count suggests — used to spot changes as the network learns.
  • developmental interpretabilitydevelopmental interpretability, n.: The study of how a neural network's internal structure forms during training — when and how new abilities and mechanisms appear — rather than only examining the finished model.
  • Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability.
  • loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys.
  • Occam's razorOccam's razor, n.: A rule of thumb for choosing between explanations: if two fit the evidence equally well, bet on the one that assumes less. In AI it motivates preferring simpler models, which are less likely to fit noise.
  • minimum description lengthminimum description length, n.: The principle that the best explanation of some data is the one that lets you write the data down most briefly, counting both the rule and its exceptions. Good theories compress.

Sources

  1. Watanabe, S. (2009). Algebraic Geometry and Statistical Learning Theory. Cambridge University Press.
  2. Watanabe, S. (2001). Algebraic Analysis for Nonidentifiable Learning Machines. Neural Computation 13(4).

Mentioned in

loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys., developmental interpretabilitydevelopmental interpretability, n.: The study of how a neural network's internal structure forms during training — when and how new abilities and mechanisms appear — rather than only examining the finished model., local learning coefficientlocal learning coefficient, n.: A number measuring how complex a trained neural network effectively is around its current settings — usually far less than its raw parameter count suggests — used to spot changes as the network learns., statistical learning theorystatistical learning theory, n.: The mathematics of when and why learning from examples works: how much data is needed, and how well a model trained on some examples will do on new ones.