FP32

The IEEE 754 single-precision floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster. (binary32): 1 sign bit, 8 exponent bits and 23 stored significand bits, giving about 7 decimal digits of precision and a range up to about 3.4 × 10^38. It was the standard format for neural-network training until the late 2010s and remains the usual format for master weights, optimizer states, accumulations, loss computation and numerically sensitive operations in mixed-precision trainingmixed-precision training, n.: Training a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster., while matrix arithmetic moves to bfloat16bfloat16, n.: A 16-bit number format invented at Google for AI. It keeps the huge range of the standard 32-bit format but drops precision, which neural networks can usually tolerate., FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits. or narrower formats. TF32, introduced with Nvidia's A100 (2020), is a tensor-core mode that takes FP32 inputs but multiplies them with only 10 significand bits — FP32's range with FP16FP16, abbr.: A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values.'s precision — and accumulates in FP32.