INT8
In plain wordsStoring numbers as whole numbers from −128 to 127 using 8 bits. AI models converted to this format run faster and need a quarter of the memory of 32-bit versions.
An 8-bit signed integer format representing 256 evenly spaced values (−128 to 127), used in AI for quantizedquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. and activations. Real values are mapped to integers with a scale factor (and optionally a zero point) chosen per tensor, per channel or per group; integer multiply–accumulates are summed in 32-bit accumulators and rescaled. INT8 arithmetic is cheaper in silicon area and energy than floating point, which made it the standard for NPUsNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud., early accelerators such as the first TPUTPU, abbr.: Google's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud., and server inference via post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding. or quantization-aware trainingquantization-aware training, n.: Training or fine-tuning an AI model while simulating the low-precision numbers it will use later, so that it learns to work well despite the rounding.. Large language models complicate it because a few activation channels carry extreme outliers, a problem addressed by methods such as LLM.int8() and SmoothQuant. INT4 (16 values) is common for weight-only quantization, while floating-point FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits. and FP4FP4, abbr.: A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory. compete with integer formats on newer hardware.
Origin
Eight-bit integer inference of neural networks was demonstrated on CPUs by Vincent Vanhoucke, Andrew Senior and Mark Mao (2011) and became mainstream with Google's integer-only inference scheme (Benoit Jacob and colleagues, 2017–2018) and hardware such as the first TPU. Tim Dettmers and colleagues extended it to large language models with LLM.int8() (2022).
Related
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding.
- quantization-aware trainingquantization-aware training, n.: Training or fine-tuning an AI model while simulating the low-precision numbers it will use later, so that it learns to work well despite the rounding.
- FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits.
- floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster.
- TOPSTOPS, abbr.: Trillions of operations per second: the speed rating usually given for AI chips in phones and laptops, often counting simple whole-number math rather than precise fractions.
- NPUNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud.
- FP8FP8, abbr.: An 8-bit number format that uses a quarter of the memory of the standard 32-bit format, letting new AI chips train and run models roughly twice as fast as with 16 bits.
Sources
- Jacob, B. et al. (2018). Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference. Proceedings of CVPR 2018.
- Dettmers, T., Lewis, M., Belkada, Y. & Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NeurIPS 2022.
Mentioned in
AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models., floating-point formatfloating-point format, n.: A way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster., NPUNPU, abbr.: A small AI chip, or part of a chip, built into phones, laptops and other devices so they can run AI features quickly without draining the battery or using the cloud., FP4FP4, abbr.: A tiny 4-bit number format that can represent only sixteen different values. With clever scaling tricks it lets the newest AI chips run, and even train, models faster and with less memory., microscaling formatsmicroscaling formats, n.: Number formats that store small groups of numbers with one shared scale factor, so each number can use very few bits while the group as a whole stays accurate., TOPSTOPS, abbr.: Trillions of operations per second: the speed rating usually given for AI chips in phones and laptops, often counting simple whole-number math rather than precise fractions.