training run
In plain wordsOne complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months.
A single execution of a training job under one configuration — architecture, initialization, data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at., token budget, hyperparametershyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data. and parallel layout — from initialization (or a starting checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run.) to its planned end. Practitioners distinguish many small ablation or proxy runs, used to choose settings and fit scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments., from the main or 'hero' run that produces the released model. As of 2026 a frontier pretraining run lasts weeks to months on tens of thousands of GPUsGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. or TPUsTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud., in the largest reported cases more than 100,000, and is restarted from checkpoints many times after hardware failures, loss spikesloss spike, n.: A sudden jump in a model's error partway through training. Sometimes the model recovers on its own; sometimes the run falls apart and engineers must rewind to an earlier saved copy. or deliberate interventions; it is still called one run because it follows one plan and one loss curveloss curve, n.: A graph of a model's error as training goes on. A healthy curve drops steeply and then keeps sloping gently downward; bumps, jumps or flat stretches can warn engineers that something is wrong.. Its scale is reported as training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules. (FLOP), accelerator-hours or cost.
We fit the scaling law on forty small runs before committing the whole cluster to the main run.
Origin
'Run' is long-standing computing usage for one execution of a program or experiment, and 'hero run' is high-performance-computing jargon for a showcase job using a whole machine. Both gained weight in AI in the early 2020s, when single frontier runs became costly, scheduled events discussed by name.
Usage note
Reported training costs usually cover only the final run, not the experiments, failed runs and staff behind it. DeepSeek, for example, stated that its roughly 5.6-million-dollar estimate for DeepSeek-V3 (December 2024) excluded prior research and ablations.
Related
- training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules.
- compute budgetcompute budget, n.: The total amount of computing a team can spend on training (or running) a model, set in advance like a money budget and used to decide how big the model should be and how much data it sees.
- checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run.
- loss curveloss curve, n.: A graph of a model's error as training goes on. A healthy curve drops steeply and then keeps sloping gently downward; bumps, jumps or flat stretches can warn engineers that something is wrong.
- YOLO runYOLO run, n.: Slang for launching a big, expensive training run without the usual careful small-scale tests first — changing many things at once and trusting experience and instinct that it will work. YOLO means 'you only live once'.
- goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough.
- pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses.
- training steptraining step, n.: One round of learning: the model makes predictions on a small batch of examples, measures its error and adjusts its settings once. Training runs consist of thousands to millions of steps.
Mentioned in
pretrainingpretraining, n.: The first and usually the most expensive stage of building a modern AI model, in which it learns general patterns from enormous amounts of raw text, images or code before being specialized for particular uses., trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better., AI supercomputerAI supercomputer, n.: A huge network of AI chips — often tens or hundreds of thousands — wired together in a data center to train the largest AI models., checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run., convergenceconvergence, n.: The point at which training settles down and further steps no longer meaningfully improve the model — and, in theory, a guarantee that a method will reach a solution, and how fast., experiment trackingexperiment tracking, n.: Keeping an automatic logbook of every model-training attempt — settings, data, code and results — so a team can compare attempts, repeat the good ones and see what changed., loss curveloss curve, n.: A graph of a model's error as training goes on. A healthy curve drops steeply and then keeps sloping gently downward; bumps, jumps or flat stretches can warn engineers that something is wrong., power lawpower law, n.: A rule linking two quantities in which scaling one up by a given factor always changes the other by the same multiple, wherever you start. Plotted on logarithmic axes, it is a straight line., tokens per parametertokens per parameter, n.: How many pieces of text a language model reads during training for each adjustable number inside it — a quick way to tell whether a model was trained on a little or a lot of data for its size., training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules., training steptraining step, n.: One round of learning: the model makes predictions on a small batch of examples, measures its error and adjusts its settings once. Training runs consist of thousands to millions of steps., goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough., model FLOPs utilizationmodel FLOPs utilization, n.: The share of a computer's top calculating speed that actually goes into useful AI training work. Real systems often reach only around a third to a half., YOLO runYOLO run, n.: Slang for launching a big, expensive training run without the usual careful small-scale tests first — changing many things at once and trusting experience and instinct that it will work. YOLO means 'you only live once'.