CPU

The general-purpose processor that fetches and executes a program's instructions, optimized for fast sequential work through a modest number of complex cores with large caches, branch prediction and out-of-order execution. Server CPUs now have dozens to a few hundred cores and vector or matrix extensions (such as AVX-512, AMX or Arm SVE) that can run small models, but their arithmetic throughput and memory bandwidthmemory bandwidth, n.: How fast data can be moved between a chip's memory and its calculators. For chatbots, this often decides how quickly words appear, more than raw calculating speed. are far below those of a GPUGPU, abbr.: A computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time. or other AI acceleratorAI accelerator, n.: Any chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models.. In AI clusters the CPU is the host: it runs the operating system and framework code, loads and preprocesses data, launches kernelskernel, n.: A word with several AI meanings: a similarity measure used by some learning methods; a small grid of numbers that scans an image inside a neural network; or a small program that runs on a GPU. on accelerators and manages networking and storage. CPU-only inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it. remains common for small or quantizedquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. models on laptops and servers without accelerators.