model FLOPs utilization
In plain wordsThe share of a computer's top calculating speed that actually goes into useful AI training work. Real systems often reach only around a third to a half.
A measure of how much of a system's peak arithmetic rate a training job (or sometimes an inference service) turns into useful model computation: model FLOPFLOP, n.: A single arithmetic step, such as multiplying or adding two numbers. Counting them measures how much computing went into an AI model, and laws use such counts to decide which models face extra rules. achieved per second divided by the hardware's theoretical peak FLOPSFLOPS, abbr.: Floating-point operations per second: how many arithmetic steps a computer can do each second. It is the standard speed rating for supercomputers and AI chips.. Only work the model strictly requires is credited, namely its forwardforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction. and backwardbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change. passes (about 6 × parametersparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned. × tokens per second for a dense transformer, plus attention), so recomputing activations for gradient checkpointinggradient checkpointing, n.: A memory-saving trick: instead of storing every intermediate result needed for learning, the computer keeps only a few and recalculates the rest when needed, trading extra computing time for memory. earns nothing and runs built on different software can be compared on equal terms. Hardware FLOPs utilization (HFU) instead counts every operation the chips execute, recomputation included. Large training runs commonly reach roughly 30–50% MFU; the shortfall comes from communication, memory-bound operations, pipeline bubbles, stragglers and failures. MFU complements wall-clock measures such as goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough..
After overlapping the all-reduce with the backward pass, MFU rose from 38% to 45%.
Origin
Proposed by Aakanksha Chowdhery and colleagues at Google in the PaLM paper (April 2022) as an implementation-independent metric; they reported 46.2% MFU (57.8% HFU) for PaLM 540B on 6,144 TPU v4 chips, against an estimated 21.3% for GPT-3.
Usage note
Peak FLOPS depends on the number format, so MFU for an FP8 run should be computed against FP8 peak; published figures are not always explicit about this.
Related
- FLOPSFLOPS, abbr.: Floating-point operations per second: how many arithmetic steps a computer can do each second. It is the standard speed rating for supercomputers and AI chips.
- goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough.
- roofline modelroofline model, n.: A simple chart showing the fastest a program can run on a chip: limited either by how quickly data arrives from memory or by the chip's top calculating speed.
- training runtraining run, n.: One complete attempt at training a model with a fixed recipe, from start to finish. For the largest AI models a single run can occupy tens of thousands of chips for months.
- distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks.
- arithmetic intensityarithmetic intensity, n.: How much calculating a task does for each piece of data it fetches from memory. Tasks with little calculation per fetch are slowed by memory, not by the chip's math speed.
Sources
- Chowdhery, A. et al. (2022). PaLM: Scaling Language Modeling with Pathways. arXiv:2204.02311. arxiv.org/abs/2204.02311
Mentioned in
FLOPFLOP, n.: A single arithmetic step, such as multiplying or adding two numbers. Counting them measures how much computing went into an AI model, and laws use such counts to decide which models face extra rules., FLOPSFLOPS, abbr.: Floating-point operations per second: how many arithmetic steps a computer can do each second. It is the standard speed rating for supercomputers and AI chips., GPU-hourGPU-hour, n.: One graphics chip working for one hour — the everyday unit for how much computing a job used and for renting AI chips by the hour., training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules., goodputgoodput, n.: The share of a big training job's time and computing that actually moves training forward, after subtracting time lost to crashes, restarts, waiting and repeated work. When serving users, it means the rate of requests answered fast enough., sequence parallelismsequence parallelism, n.: Splitting the long string of words a model is working on into chunks handled by different chips, which saves memory and makes it possible to train on very long documents., stragglerstraggler, n.: One slow computer or chip in a large group working in lockstep. Because everyone must wait for the slowest member at each step, a single straggler can slow down an entire AI training job.