offloading

Memory

The keeping of some of a model's weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself., KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch., activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. or optimizer states in a slower, larger memory tier (CPU RAM, NVMe SSD or remote storage) and moving them into accelerator memory only when needed, so that models or contexts larger than GPU memory can run. Offloading trades speed for capacity: transfers over PCIe are far slower than high-bandwidth memoryhigh-bandwidth memory, n.: Very fast memory made by stacking memory chips on top of each other and placing them right next to the processor, so AI chips can be fed data quickly enough., so systems overlap them with computation, prefetch the next layer's weights, compress what they move, or run some layers on the CPU. It underlies running large models on consumer hardware (for example, putting only some layers on the GPU in llama.cppllama.cpp, prop. n.: A free, open-source program that runs AI language models on ordinary computers and phones, without needing a powerful graphics card, by using compact low-precision versions of the models.), throughput-oriented batch inferencebatch inference, n.: Running an AI model over a big pile of inputs as one job rather than answering each request live, like processing a stack of forms overnight. Results come back later but cost less. on a single GPU, tiered KV-cache storage in disaggregated servingdisaggregated serving, n.: Running the two stages of an AI's reply, reading the prompt and writing the answer, on separate groups of chips, each tuned for its own job, and handing the work between them., and memory-saving training methods such as ZeRO-Offload.

With 24 GB of VRAM we offload twenty layers to system RAM and still get eight tokens per second.

Edge computing

Sending computation from a resource-constrained device such as a phone, sensor or vehicle to a nearby edge server or the cloud, trading network delay, privacy and connectivity against on-device energy and compute; edge AIedge AI, n.: Doing AI's computing on or near the devices that collect the data, such as cameras, cars, factory machines, phones or local servers, instead of sending everything off to distant data centers. systems may also split one model between device and server.