data wall

The projected point at which scalingscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments. of pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses. is limited by the finite stock of high-quality human-generated data rather than by compute. Compute-optimal training (Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model.) calls for roughly 20 tokens per parameter, leading models already train on tens of trillions of tokens, and estimates put the stock of usable public text at a few hundred trillion tokens; Epoch AI researchers projected in 2024 that training sets would match that stock between 2026 and 2032, earlier if models are overtrainedovertraining, n.: Deliberately training a model on much more data than would be most efficient for its training budget, so that a smaller model reaches good quality and is cheaper to run afterwards.. Proposed ways past the wall include repeating data for several epochs (data-constrained scalingdata-constrained scaling, n.: How to keep improving AI models when the supply of fresh, high-quality training text runs out — for example by reusing the same data several times, which works well at first and then stops helping.), better filtering, multimodal and private data, synthetic datasynthetic data, n.: Data created by computer programs or AI models rather than collected from the real world — such as practice math problems written by a model, or simulated street scenes for self-driving cars., reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. with verifiable rewards and self-play, and shifting gains to test-time computetest-time compute, n.: The computing effort an AI model spends while answering a question, as opposed to while being trained. Letting a model think longer or try many answers can make it markedly smarter.. Whether and when the wall binds is contested: optimists cite synthetic data and new sources, skeptics diminishing returns and model collapsemodel collapse, n.: The gradual decline of AI models trained on the output of earlier AI models: rare details vanish first, then quality and variety erode, like a photocopy of a photocopy..

If the data wall is real, the next gains will come from RL and synthetic data, not bigger crawls.