GPTQ
In plain wordsA popular method for shrinking large language models to about 4 bits per number after training, carefully adjusting the remaining numbers to make up for each rounding error.
A one-shot post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding. method for large transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. that reduces weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. to 2–4 bits while compensating for the error introduced. Working one layer at a time on a small calibration set, it quantizes a weight matrix column by column and, after each column is rounded, updates the not-yet-quantized columns to offset the change in the layer's output, using the inverse of a HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass. approximated from the layer's inputs. With lazy batched updates and a Cholesky formulation, its authors quantized a 175-billion-parameter model in about four GPU hours, to 3 or 4 bits with little loss in perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll.; the result fit on a single GPU, and their custom kernels generated text roughly 3.25–4.5 times faster than FP16. Its group-wise 4-bit checkpoints became a common distribution format, supported by vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models., Hugging Face Transformers and other engines.
The GPTQ 4-bit checkpoint with group size 128 runs comfortably on a single 48 GB card.
Origin
Introduced by Elias Frantar, Saleh Ashkboos, Torsten Hoefler and Dan Alistarh of IST Austria and ETH Zurich (arXiv, October 2022; ICLR 2023), scaling up their earlier Optimal Brain Quantization, itself descended from the Optimal Brain Surgeon pruning method (1993). The name joins GPT with Q for quantization.
Related
- post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding.
- AWQAWQ, abbr.: A method for shrinking large language models to about 4 bits per number that protects the small share of numbers most important to the model, judged by the data flowing through them.
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass.
- GGUFGGUF, abbr.: A file format that packs an AI language model, often in a shrunken low-precision version, into one file that ordinary computers can load and run quickly; used by llama.cpp and Ollama.
- SmoothQuantSmoothQuant, prop. n.: A method that lets large language models run entirely on 8-bit numbers by moving hard-to-round extreme values out of the data flowing through the model and into its fixed numbers, which cope better.
- AWQAWQ, abbr.: A method for shrinking large language models to about 4 bits per number that protects the small share of numbers most important to the model, judged by the data flowing through them.
Sources
- Frantar, E., Ashkboos, S., Hoefler, T. & Alistarh, D. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. arXiv:2210.17323 (ICLR 2023). arxiv.org/abs/2210.17323
Mentioned in
post-training quantizationpost-training quantization, n.: Shrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding., quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., vLLMvLLM, prop. n.: A free, open-source program for running large language models on servers efficiently, so one machine can answer many more users at once. Many companies use it to host AI models., AWQAWQ, abbr.: A method for shrinking large language models to about 4 bits per number that protects the small share of numbers most important to the model, judged by the data flowing through them., LLM.int8()LLM.int8(), prop. n.: An early, widely used method for running very large language models with 8-bit numbers, roughly halving their memory needs, by keeping a few unusually large values at higher precision.