parameter count
In plain wordsHow many adjustable numbers an AI model contains — the usual measure of its size. Modern language models have billions to trillions of them.
The number of trainable values (weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. and biases) in a neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain., the conventional measure of model size, written with B for billions or T for trillions ('a 70B model'). It determines memory needs — about 2 bytes per parameter at 16-bit precision, so a 70B model needs roughly 140 GB before quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy. — and, together with the number of training tokens, training compute (about 6 × parameters × tokens FLOPsFLOP, n.: A single arithmetic step, such as multiplying or adding two numbers. Counting them measures how much computing went into an AI model, and laws use such counts to decide which models face extra rules. for dense transformers). In mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network. models, total parameters (all experts) differ from the active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests. used for each token: DeepSeek-V3, for example, has 671 billion in total but about 37 billion active. Counts are imperfect proxies for capability, since data quantity and quality, training compute and post-training matter as much, and scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments. sometimes exclude embedding parameters. Several leading developers have not disclosed parameter counts for their frontier models since 2023.
Despite its smaller parameter count, the 8B model beats last year's 70B on most benchmarks.
Origin
Counting parameters is as old as neural networks. For language models the figure became a headline number with GPT-2 (1.5 billion parameters, 2019) and GPT-3 (175 billion, 2020), and the trade-off between parameters and training tokens was formalized in the Kaplan (2020) and ChinchillaChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model. (2022) scaling laws.
Usage note
Do not confuse parameters (learned values) with hyperparametershyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data. (settings chosen by engineers) or with context windowcontext window, n.: How much an AI model can read and keep in view at once: the instructions, the conversation so far and any attached documents. Text that does not fit has to be cut, summarized or left out. length, which is also quoted as a number.
Related
- parameterparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned.
- weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself.
- scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
- mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
- active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests.
- quantizationquantization, n.: Storing an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
- small language modelsmall language model, n.: A language model built to be compact enough to run cheaply and quickly, even on a laptop or phone, trading some knowledge and skill for speed, privacy and lower cost.
- Chinchilla scaling lawsChinchilla scaling laws, n.: A 2022 DeepMind finding that, for a fixed amount of computing, language models had been made too big and trained on too little text; the best balance is roughly 20 word-pieces of text for every adjustable number inside the model.
- hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data.
Mentioned in
active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests., mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network., model capacitymodel capacity, n.: How much a model is able to learn: the variety and complexity of patterns it can represent or memorize. Too little and it misses real patterns; very large models can even memorize pure noise., parameterparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned., small language modelsmall language model, n.: A language model built to be compact enough to run cheaply and quickly, even on a laptop or phone, trading some knowledge and skill for speed, privacy and lower cost., tokens per parametertokens per parameter, n.: How many pieces of text a language model reads during training for each adjustable number inside it — a quick way to tell whether a model was trained on a little or a lot of data for its size., overparameterizationoverparameterization, n.: Giving a model far more adjustable settings than it would seem to need — often more than it has training examples. Classic theory predicted this would backfire; in deep learning it usually helps.