algorithmic progress
In plain wordsImprovements in AI methods — not faster chips — that let a model reach the same ability with less computing power. Researchers estimate this has cut the computing needed by half every eight to sixteen months.
Reductions over time in the compute (or data) required to reach a fixed level of performance, attributable to better architectures, training objectives, optimizers, data and recipes rather than better hardware. It is usually measured as a halving time for the compute needed to hit a benchmark score or loss, or equivalently as a compute multipliereffective compute, n.: A way of measuring AI progress that counts better methods as if they were extra computing power: if a new technique lets a model match one trained with twice the computing, it counts as doubling effective compute. relative to an earlier method. Danny Hernandez and Tom Brown (2020) found that reaching AlexNet-level ImageNet accuracy took 44 times less compute in 2019 than in 2012, a halving every 16 months. Anson Ho and colleagues (2024) estimated that the compute needed for a given language-model performance halved about every 8 months from 2012 to 2023 (95% interval roughly 5–14 months), faster than Moore's lawMoore's law, n.: Gordon Moore's prediction that chipmakers could pack twice as many transistors onto a chip roughly every two years — a pace that for decades made computers steadily faster and cheaper, and has slowed but not stopped., though growth in compute contributed more to overall gains. Estimates are sensitive to the benchmark chosen, and data-quality and post-training gains are hard to separate from scale.
If algorithmic progress continues at this rate, next year's model will need half the compute for the same loss.
Origin
Katja Grace's 'Algorithmic Progress in Six Domains' (Machine Intelligence Research Institute, 2013) was an early systematic study, comparing algorithmic and hardware gains in areas such as chess and SAT solving. Hernandez and Brown's OpenAI analysis (2020) and Epoch AI's estimates from 2022 made it a standard input to AI forecasting.
Usage note
Distinct from unhobblingunhobbling, n.: Fixing the simple ways a capable AI model is held back — such as by giving it tools, memory or time to think — so that abilities it already has can actually be used. and post-training gains that unlock latent abilities without reducing pretraining compute, which some analysts count separately. Halving times describe past trends, not guaranteed future rates.
Related
- effective computeeffective compute, n.: A way of measuring AI progress that counts better methods as if they were extra computing power: if a new technique lets a model match one trained with twice the computing, it counts as doubling effective compute.
- training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules.
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
- Moore's lawMoore's law, n.: Gordon Moore's prediction that chipmakers could pack twice as many transistors onto a chip roughly every two years — a pace that for decades made computers steadily faster and cheaper, and has slowed but not stopped.
- AI timelinesAI timelines, n.: Forecasts of when AI will reach major milestones, such as matching humans at most work. People with short timelines expect this within years; others expect decades, or never.
- Epoch AIEpoch AI, prop. n.: A research nonprofit that gathers data on how fast AI is advancing — the computing power, data and money behind it — and publishes forecasts and difficult tests for AI.
- the bitter lessonthe bitter lesson, phr.: The argument, made by AI pioneer Rich Sutton in 2019, that AI has advanced mostly through general methods that use ever more computing power, not by programming in human knowledge.
Sources
- Hernandez, D. & Brown, T. B. (2020). Measuring the Algorithmic Efficiency of Neural Networks. arXiv:2005.04305. arxiv.org/abs/2005.04305
- Ho, A., Besiroglu, T., Erdil, E., Owen, D., Rahman, R., Guo, Z. C., Atkinson, D., Thompson, N. & Sevilla, J. (2024). Algorithmic progress in language models. arXiv:2403.05812. arxiv.org/abs/2403.05812
- Grace, K. (2013). Algorithmic Progress in Six Domains. Machine Intelligence Research Institute technical report.
Mentioned in
Jevons paradoxJevons paradox, n.: The observation that making something more efficient can make people use more of it overall because it gets cheaper — as with coal in the 1800s, and with AI computing today., effective computeeffective compute, n.: A way of measuring AI progress that counts better methods as if they were extra computing power: if a new technique lets a model match one trained with twice the computing, it counts as doubling effective compute., Epoch Capabilities IndexEpoch Capabilities Index, prop. n.: A single number summarizing an AI model's general capability, calculated by the research group Epoch AI by combining its scores on dozens of tests onto one common scale., unhobblingunhobbling, n.: Fixing the simple ways a capable AI model is held back — such as by giving it tools, memory or time to think — so that abilities it already has can actually be used.