LLM.int8()
In plain wordsAn early, widely used method for running very large language models with 8-bit numbers, roughly halving their memory needs, by keeping a few unusually large values at higher precision.
A method for running transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. language models with 8-bit integer matrix multiplication without losing accuracy. Simple 8-bit schemes fail on large models because a handful of hidden-state dimensions carry extreme values, 'outlier features' that its authors showed appear systematically once models pass several billion parameters. LLM.int8() deals with them in two parts: each row and column entering a matrix product gets its own scaling constant (vector-wise quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.), and the few outlier dimensions are pulled out and multiplied in 16-bit precision, a tiny fraction of the work, while everything else uses 8-bit integers. This halved inference memory with no measurable degradation up to 175 billion parameters, enough to fit the largest open models of 2022, such as OPT-175B and BLOOM, onto one machine with consumer graphics cards. Its integration into the bitsandbytes library and Hugging Face Transformers made 8-bit loading routine and led directly to 4-bit fine-tuning with QLoRAQLoRA, n.: A memory-saving version of LoRA that squeezes the big model into a compact 4-bit form while training small add-ons, letting very large models be customized on a single graphics card..
Load the model with LLM.int8() if your GPU has only half the memory the 16-bit weights need.
Origin
Introduced by Tim Dettmers, Mike Lewis, Younes Belkada and Luke Zettlemoyer, from the University of Washington, Meta AI and Hugging Face (arXiv, August 2022; NeurIPS 2022). The name is styled as a method call that casts an LLM to 8-bit integers.
Usage note
The outlier features it documented became a central concern of later methods such as SmoothQuantSmoothQuant, prop. n.: A method that lets large language models run entirely on 8-bit numbers by moving hard-to-round extreme values out of the data flowing through the model and into its fixed numbers, which cope better. and AWQAWQ, abbr.: A method for shrinking large language models to about 4 bits per number that protects the small share of numbers most important to the model, judged by the data flowing through them., and of interpretability work on unusually large activations.
Related
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding.
- SmoothQuantSmoothQuant, prop. n.: A method that lets large language models run entirely on 8-bit numbers by moving hard-to-round extreme values out of the data flowing through the model and into its fixed numbers, which cope better.
- QLoRAQLoRA, n.: A memory-saving version of LoRA that squeezes the big model into a compact 4-bit form while training small add-ons, letting very large models be customized on a single graphics card.
- GPTQGPTQ, prop. n.: A popular method for shrinking large language models to about 4 bits per number after training, carefully adjusting the remaining numbers to make up for each rounding error.
Sources
- Dettmers, T., Lewis, M., Belkada, Y. & Zettlemoyer, L. (2022). LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale. NeurIPS 2022. arXiv:2208.07339. arxiv.org/abs/2208.07339
Mentioned in
post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding., SmoothQuantSmoothQuant, prop. n.: A method that lets large language models run entirely on 8-bit numbers by moving hard-to-round extreme values out of the data flowing through the model and into its fixed numbers, which cope better.