llama.cpp

An open-source program for running large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. on ordinary computers, written in plain C and C++ on top of Georgi Gerganov's ggml tensor library so that it builds almost anywhere with few dependencies. Models are packaged in its GGUFGGUF, abbr.: A file format that packs an AI language model, often in a shrunken low-precision version, into one file that ordinary computers can load and run quickly; used by llama.cpp and Ollama. file format and usually compressed by integer quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. to between about 1.5 and 8 bits per weight, so that large models fit in a laptop's memory; one code base targets ordinary CPUs, Apple's Metal, NVIDIA and AMD graphics cards and cross-vendor interfaces such as Vulkan, and it can keep some layers on the CPU while the rest run on a GPU (offloadingoffloading, n.: Moving part of an AI model's data or work out of a chip's scarce fast memory onto ordinary memory, a disk or another computer, so that models too big for the chip can still run.). Besides a library and command-line tools it includes llama-server, an HTTP server with an OpenAI-compatible APIOpenAI-compatible API, n.: An AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key., continuous batchingcontinuous batching, n.: A way for an AI server to handle many conversations together, letting new ones join and finished ones leave after every word, instead of waiting for a whole group to finish., speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality. and grammar-constrained decodingconstrained decoding, n.: Forcing an AI model's output to follow a required format — such as valid code or a fill-in form — by blocking, at every step, any word that would break the rules.. It runs text, vision-language and other model families, and is the engine beneath many desktop and mobile apps, including OllamaOllama, prop. n.: A free app that makes it easy to download and run AI language models on your own computer with a single command, keeping your data on your machine., making it central to on-device AIon-device AI, n.: AI that runs directly on your own phone, laptop, car or headset rather than on a company's distant servers, so it can work offline, respond instantly and keep your data on the device. with open-weight modelsopen-weight model, n.: An AI model whose trained internal numbers are published for anyone to download and run on their own computers, even if the data and code used to make it stay private..

I converted the model to GGUF and ran it with llama.cpp on a Raspberry Pi, slowly but fully offline.