model collapse
In plain wordsThe gradual decline of AI models trained on the output of earlier AI models: rare details vanish first, then quality and variety erode, like a photocopy of a photocopy.
Progressive loss of fidelity in generative models that are trained, generation after generation, on content produced by their predecessors, so that each new model represents the real data distribution less faithfully than the last. Because each generation samples from, and slightly misestimates, the previous model's distribution, low-probability events are underrepresented and then lost. Researchers describe two stages: first the rare cases at the edges of the distribution thin out, removing unusual and minority content while typical outputs still look fine; eventually outputs narrow into a distorted, low-variety distribution that has drifted far from the original. The effect has been demonstrated for language models, variational autoencoders, Gaussian mixture models and image generators. Its practical importance is debated: experiments in which synthetic data accumulates alongside the original human data, rather than replacing it, avoid collapse, and curated, verified synthetic datasynthetic data, n.: Data created by computer programs or AI models rather than collected from the real world — such as practice math problems written by a model, or simulated street scenes for self-driving cars. is widely used to improve models. The concern persists because a growing share of web text is AI-generated (see AI slopAI slop, n.: Low-quality content churned out in bulk by AI — bland articles, fake-looking images, filler videos — posted to grab clicks or fill space, with little care for accuracy or value.).
After five generations trained only on their own outputs, the models had forgotten every rare dog breed.
Origin
Named by Ilia Shumailov, Zakhar Shumaylov, Yiren Zhao, Yarin Gal, Nicolas Papernot and Ross Anderson in 'The Curse of Recursion: Training on Generated Data Makes Models Forget' (arXiv, May 2023), published in Nature in July 2024. Sina Alemohammad and colleagues described the same effect as 'Model Autophagy Disorder' (2023).
Usage note
Not to be confused with mode collapsemode collapse, n.: When a generator gets stuck producing only a few kinds of output — the same face, the same joke — instead of the full variety it was meant to learn. in generative adversarial networks, where a generator fails to cover the data's variety during training, or catastrophic forgettingcatastrophic forgetting, n.: When a neural network trained on something new gets suddenly and badly worse at what it could already do, because the new training rewrites the internal settings the old skills relied on., the loss of old skills during fine-tuning.
Related
- synthetic datasynthetic data, n.: Data created by computer programs or AI models rather than collected from the real world — such as practice math problems written by a model, or simulated street scenes for self-driving cars.
- AI slopAI slop, n.: Low-quality content churned out in bulk by AI — bland articles, fake-looking images, filler videos — posted to grab clicks or fill space, with little care for accuracy or value.
- long taillong tail, n.: The huge number of rare things — uncommon words, unusual road situations, obscure facts — that each appear seldom but together make up a large share of what an AI system meets.
- data provenancedata provenance, n.: The documented history of a piece of data: where it came from, who created and changed it, under what permissions, and how it was processed before reaching an AI system.
- data walldata wall, n.: The worry that AI companies will run out of new human-written text to train ever-larger models on, because the useful public internet has already been largely used up.
- mode collapsemode collapse, n.: When a generator gets stuck producing only a few kinds of output — the same face, the same joke — instead of the full variety it was meant to learn.
- catastrophic forgettingcatastrophic forgetting, n.: When a neural network trained on something new gets suddenly and badly worse at what it could already do, because the new training rewrites the internal settings the old skills relied on.
Sources
- Shumailov, I., Shumaylov, Z., Zhao, Y., Papernot, N., Anderson, R. & Gal, Y. (2024). AI Models Collapse When Trained on Recursively Generated Data. Nature 631, 755–759.
- Shumailov, I. et al. (2023). The Curse of Recursion: Training on Generated Data Makes Models Forget. arXiv:2305.17493. arxiv.org/abs/2305.17493
- Gerstgrasser, M. et al. (2024). Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Data. arXiv preprint.
Mentioned in
AI slopAI slop, n.: Low-quality content churned out in bulk by AI — bland articles, fake-looking images, filler videos — posted to grab clicks or fill space, with little care for accuracy or value., synthetic datasynthetic data, n.: Data created by computer programs or AI models rather than collected from the real world — such as practice math problems written by a model, or simulated street scenes for self-driving cars., data walldata wall, n.: The worry that AI companies will run out of new human-written text to train ever-larger models on, because the useful public internet has already been largely used up., dead internet theorydead internet theory, n.: The idea, first a fringe conspiracy theory, that much of what we see online is now made by bots and AI rather than by real people., feedback loopfeedback loop, n.: A cycle in which what a system produces comes back to shape its own future input, like a microphone howling near its loudspeaker. In AI, a model's predictions can shape the very data it later learns from., homogenizationhomogenization, n.: The tendency of AI to make things more alike: products built on the same few models share the same flaws, and people who write or create with AI produce more similar work., long taillong tail, n.: The huge number of rare things — uncommon words, unusual road situations, obscure facts — that each appear seldom but together make up a large share of what an AI system meets., mode collapsemode collapse, n.: When a generator gets stuck producing only a few kinds of output — the same face, the same joke — instead of the full variety it was meant to learn., self-improvementself-improvement, n.: When an AI gets better by learning from material it produces itself — its own answers, critiques or practice problems — rather than only from new human-made examples., blurry JPEG of the webblurry JPEG of the web, phr.: Writer Ted Chiang's 2023 comparison of ChatGPT to a low-quality compressed photo of the internet: it keeps the gist of what it learned but blurs the details and fills gaps with plausible guesses., entropy collapseentropy collapse, n.: A failure in reinforcement-learning training where an AI quickly becomes overconfident and stops trying different answers, so it explores less and its improvement stalls.