grokking
In plain wordsWhen a model that has merely memorized its training examples suddenly, long afterward, figures out the underlying rule and starts getting new examples right.
A training phenomenon in which a neural network first fits its training data perfectly while performing at chance on held-out data — apparent memorization — and then, after many more optimization steps (often orders of magnitude more), abruptly generalizes to near-perfect test accuracy. It was first reported on small algorithmic tasks such as modular arithmetic and permutation-group operations and has since been induced on other tasks under suitable conditions. Weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data. strongly promotes it, and the smaller the training set, the longer the delay before it occurs. Mechanisticmechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior. analysis of transformers trained on modular addition showed the network gradually building a generalizing circuit, based on discrete Fourier transforms and trigonometric identities, while memorizing, then removing the memorized solution, so the sudden jump in test accuracy hides continuous internal progress. Explanations stress competition between memorizing and generalizing solutions that differ in weight norm and efficiency. Grokking undermines naive early stoppingearly stopping, n.: Stopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples. and serves as a model of sudden capability gains.
Train accuracy hit 100% by step 1,000, but the model only grokked modular division after 100,000 steps.
Origin
Named by Alethea Power, Yuri Burda, Harri Edwards, Igor Babuschkin and Vedant Misra of OpenAI in 'Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets', presented at an ICLR 2021 workshop (May 2021) and posted to arXiv in January 2022. 'Grok', meaning to understand intuitively and completely, was coined by Robert A. Heinlein in the novel Stranger in a Strange Land (1961). Neel Nanda and colleagues reverse-engineered the mechanism in 2023.
Usage note
Often invoked by analogy for sudden capability jumps in large models, but whether those share its mechanism is unclear; see emergent abilitiesemergent abilities, n.: Skills that seem to switch on suddenly as AI models grow: small versions fail at them, then larger ones past a certain size succeed. Some researchers argue the suddenness is partly an artifact of how skills are measured..
Related
- memorizationmemorization, n.: When an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns.
- weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data.
- mechanistic interpretabilitymechanistic interpretability, n.: A branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
- double descentdouble descent, n.: A surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse.
- emergent abilitiesemergent abilities, n.: Skills that seem to switch on suddenly as AI models grow: small versions fail at them, then larger ones past a certain size succeed. Some researchers argue the suddenness is partly an artifact of how skills are measured.
- early stoppingearly stopping, n.: Stopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples.
- developmental interpretabilitydevelopmental interpretability, n.: The study of how a neural network's internal structure forms during training — when and how new abilities and mechanisms appear — rather than only examining the finished model.
Sources
- Power, A., Burda, Y., Edwards, H., Babuschkin, I. & Misra, V. (2022). Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets. arXiv:2201.02177 (first presented at the 1st Mathematical Reasoning in General Artificial Intelligence Workshop, ICLR 2021). arxiv.org/abs/2201.02177
- Nanda, N., Chan, L., Lieberum, T., Smith, J. & Steinhardt, J. (2023). Progress measures for grokking via mechanistic interpretability. ICLR 2023. arxiv.org/abs/2301.05217
Mentioned in
emergenceemergence, n.: When a whole shows properties or abilities that none of its parts has and that are hard to predict from them — like a traffic jam arising from individual cars, or new skills appearing in larger AI models., convergenceconvergence, n.: The point at which training settles down and further steps no longer meaningfully improve the model — and, in theory, a guarantee that a method will reach a solution, and how fast., early stoppingearly stopping, n.: Stopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples., emergent abilitiesemergent abilities, n.: Skills that seem to switch on suddenly as AI models grow: small versions fail at them, then larger ones past a certain size succeed. Some researchers argue the suddenness is partly an artifact of how skills are measured., GrokGrok, prop. n.: The AI chatbot and model family of Elon Musk's company xAI, launched in 2023 and built into the social network X. It is known for its irreverent style and for repeated controversies over its outputs., weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data., developmental interpretabilitydevelopmental interpretability, n.: The study of how a neural network's internal structure forms during training — when and how new abilities and mechanisms appear — rather than only examining the finished model., minimum description lengthminimum description length, n.: The principle that the best explanation of some data is the one that lets you write the data down most briefly, counting both the rule and its exceptions. Good theories compress., Neel NandaNeel Nanda, prop. n.: A British AI researcher who leads mechanistic interpretability at Google DeepMind and built TransformerLens, a widely used tool for looking inside language models., toy modeltoy model, n.: A deliberately tiny, simplified AI model trained on an artificial task, small enough for researchers to understand completely and use to test ideas about how big models work.