Skip to content
Last Dictionary
Plane
Training & Post-trainingHardware & ComputeAI Engineering & MLOps

goodput

nounAlso: training goodput, ML goodput, effective training time

In plain wordsThe share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough.

  1. 1

    Large-scale training

    The rate or fraction of useful training progress a job actually achieves, as opposed to raw hardware throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second.. It discounts time lost to scheduling and startup, hardware failures and the work redone after rolling back to the last checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run., checkpoint writing, stragglers, communication stalls and inefficient execution. Google's 'ML Productivity Goodput' (2024) combines scheduling, runtime and program goodput, the last essentially model FLOPs utilizationmodel FLOPs utilization, n.: The share of a computer's top calculating speed that actually goes into useful AI training work. Real systems often reach only around a third to a half.. Goodput is the key reliability metric for frontier runs on tens of thousands of accelerators, where some component fails every few hours: Meta reported 466 job interruptions in a 54-day period of Llama 3 pretraining, mostly traced to hardware, while keeping effective training time above 90%. Raising it involves fast failure detection, hot spares, asynchronous and in-memory checkpointing and elastic restarts.

    Async checkpointing raised our goodput from 82% to 94% on the 16,000-GPU run.

  2. 2

    Statistical efficiency

    In the Pollux scheduler (Aurick Qiao and colleagues, 2020), system throughput multiplied by statistical efficiency, which falls as the batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step. grows beyond what the optimizer can use well; the scheduler maximizes it by jointly choosing each job's resources and batch size.

  3. 3

    Model serving

    In model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost., the highest request rate a system can sustain while meeting its service-level objectivesservice-level objective, n.: A specific, measurable target for how well a service should perform — for example, 99.9 percent of requests answered within two seconds — used to decide when engineers must act., for example keeping time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear. and time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow. under set limits for 90% of requests. Unlike raw throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second., it gives no credit for responses that arrive too late. The DistServe system (Yinmin Zhong and colleagues, 2024) was designed to maximize goodput per GPU, and serving research commonly reports it.

Origin

A networking term for useful application-level data delivered per unit time, excluding retransmissions and overhead, in use by at least the 1990s. Qiao and colleagues brought it to deep-learning scheduling in Pollux (arXiv 2020; OSDI 2021); DistServe (January 2024) made it the target metric for language-model serving, and Google promoted 'ML Productivity Goodput' for large training jobs in April 2024.

Related

See also
  • model FLOPs utilizationmodel FLOPs utilization, n.: The share of a computer's top calculating speed that actually goes into useful AI training work. Real systems often reach only around a third to a half.
  • checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run.
  • training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months.
  • distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks.
  • critical batch sizecritical batch size, n.: The largest number of examples a model can usefully learn from in each training step. Below it, bigger batches let many chips work in parallel and finish sooner; above it, the extra chips are mostly wasted.
  • model servingmodel serving, n.: Running a trained AI model as a live service, so that apps and people can send it requests and get answers back quickly, reliably and at a sensible cost.
  • time to first tokentime to first token, n.: How long a chatbot takes to start answering: the pause between pressing enter and seeing the first word appear.
Contrast with
  • throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second.

Sources

  1. Qiao, A., Choe, S. K., Subramanya, S. J., Neiswanger, W., Ho, Q., Zhang, H., Ganger, G. R. & Xing, E. P. (2021). Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learning. OSDI 2021. arXiv:2008.12260. arxiv.org/abs/2008.12260
  2. Grattafiori, A. et al. (2024). The Llama 3 Herd of Models. arXiv:2407.21783.
  3. Zhong, Y., Liu, S., Chen, J., Hu, J., Zhu, Y., Liu, X., Jin, X. & Zhang, H. (2024). DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving. OSDI 2024. arXiv:2401.09670. arxiv.org/abs/2401.09670

Mentioned in

AI supercomputerAI supercomputer, n.: A huge network of AI chips — often tens or hundreds of thousands — wired together in a data center to train the largest AI models., checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run., distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks., throughputthroughput, n.: How much work an AI system gets done per second across all its users at once — for a chatbot service, roughly how much text it can produce in total each second., training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules., training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months., disaggregated servingdisaggregated serving, n.: Running the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them., model FLOPs utilizationmodel FLOPs utilization, n.: The share of a computer's top calculating speed that actually goes into useful AI training work. Real systems often reach only around a third to a half., NCCLNCCL, prop. n.: NVIDIA's software library that lets its graphics chips exchange data with each other quickly — inside one computer and across networks — and that nearly all large-scale AI training on NVIDIA hardware relies on., silent data corruptionsilent data corruption, n.: When computer hardware quietly produces wrong numbers without crashing or raising any alarm. In huge AI training runs on thousands of chips, such hidden errors can damage a model before anyone notices., stragglerstraggler, n.: One slow computer or chip in a large group working in lockstep. Because everyone must wait for the slowest member at each step, a single straggler can slow down an entire AI training job., time per output tokentime per output token, n.: Once an AI has started replying, the average wait between one piece of its answer and the next. It sets how quickly the text seems to flow.