GGUF
In plain wordsA file format that packs an AI language model, often in a shrunken low-precision version, into one file that ordinary computers can load and run quickly; used by llama.cpp and Ollama.
The single-file model format of the ggml ecosystem, read by llama.cppllama.cpp, prop. n.: A free, open-source program that runs AI language models on ordinary computers and phones, without needing a powerful graphics card, by using compact low-precision versions of the models., OllamaOllama, prop. n.: A free app that makes it easy to download and run AI language models on your own computer with a single command, keeping your data on your machine. and other local inference programs. Everything needed to run the model travels in one binary file: a header beginning with the magic bytes 'GGUF' and a version number; typed key–value metadata describing the architecture, hyperparameters, tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text. vocabulary and chat templatechat template, n.: The exact text layout a chat AI was trained on for marking who is speaking — system, user or assistant. Software must reproduce it precisely, or the model performs noticeably worse.; a table of tensor descriptors; and aligned tensor data laid out so that it can be memory-mapped and loaded almost instantly. Tensors can be stored at many precisions, from 32-bit floats to block-wise quantizedquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. types of roughly 1.5 to 8 bits per weight, named for example Q8_0, Q4_K_M or IQ2_XXS. Converting a model from PyTorch or safetensorssafetensors, n.: A file format for storing an AI model's learned numbers that cannot hide harmful programs and loads very quickly, replacing older formats that could run code when opened. weights to GGUF and choosing a quantization type is the usual first step in running it locally.
For the 7B model, the Q4_K_M GGUF is about 4 GB; the Q8_0 file is nearly twice that.
Origin
Introduced by Georgi Gerganov and contributors to llama.cpp in August 2023, replacing the earlier GGML, GGMF and GGJT formats, which could not record a model's architecture and broke whenever hyperparameters were added. The specification gives no official expansion of the letters; glosses such as 'GPT-Generated Unified Format' are unofficial.
Related
- llama.cppllama.cpp, prop. n.: A free, open-source program that runs AI language models on ordinary computers and phones, without needing a powerful graphics card, by using compact low-precision versions of the models.
- OllamaOllama, prop. n.: A free app that makes it easy to download and run AI language models on your own computer with a single command, keeping your data on your machine.
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- safetensorssafetensors, n.: A file format for storing an AI model's learned numbers that cannot hide harmful programs and loads very quickly, replacing older formats that could run code when opened.
- on-device AIon-device AI, n.: AI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device.
- ONNXONNX, prop. n.: A shared file format for trained AI models, so that a model built with one toolkit can be moved to and run by another, much as a PDF opens on any computer.
- safetensorssafetensors, n.: A file format for storing an AI model's learned numbers that cannot hide harmful programs and loads very quickly, replacing older formats that could run code when opened.
Sources
- ggml project. GGUF specification (docs/gguf.md). github.com/ggml-org/ggml/blob/master/docs/gguf.md
- Gerganov, G. et al. (2023). gguf: llama.cpp pull request #2398. github.com/ggml-org/llama.cpp/pull/2398
Mentioned in
llama.cppllama.cpp, prop. n.: A free, open-source program that runs AI language models on ordinary computers and phones, without needing a powerful graphics card, by using compact low-precision versions of the models., OllamaOllama, prop. n.: A free app that makes it easy to download and run AI language models on your own computer with a single command, keeping your data on your machine., quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy., GPTQGPTQ, prop. n.: A popular method for shrinking large language models to about 4 bits per number after training, carefully adjusting the remaining numbers to make up for each rounding error., ONNXONNX, prop. n.: A shared file format for trained AI models, so that a model built with one toolkit can be moved to and run by another, much as a PDF opens on any computer., safetensorssafetensors, n.: A file format for storing an AI model's learned numbers that cannot hide harmful programs and loads very quickly, replacing older formats that could run code when opened.