sparsity
In plain wordsThe property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation.
The fraction of elements in a vector, matrix or tensor that are zero (or of units that are inactive), and by extension the techniques that exploit it. In AI it takes several forms: weight sparsity, produced by pruningpruning, n.: Cutting away the parts of an AI model that contribute least, as a gardener trims branches, so the model becomes smaller or faster while still working nearly as well. or L1 regularization (lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model.); activation sparsity, as when ReLU units output many zeros or a sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept. is trained so that few features fire; routed or conditional sparsity, where a mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network. activates only some experts per token (see active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests.); attention sparsity, where each token attends to only some others (sparse attentionsparse attention, n.: Attention in which each word looks at only a selected subset of the other words instead of all of them, so very long texts can be processed at a fraction of the cost.); and sparse data such as one-hot or bag-of-words features. Sparsity saves memory and arithmetic only if storage formats, kernels and hardware can skip the zeros: unstructured patterns at moderate sparsity often run no faster on GPUs, which is why structured and N:M sparsityN:M sparsity, n.: A tidy pattern of zeros inside an AI model, such as two zeros in every group of four numbers, that special chip hardware can skip to run the model faster. patterns are favored. Sparse representations are also valued for interpretability and statistical efficiency.
At 90 percent unstructured sparsity the model is a tenth of the size, but our GPU kernels barely speed it up.
Origin
A mathematical idea long used in numerical computing, where specialized sparse-matrix methods date to the 1950s–60s, and the term 'sparse matrix' is sometimes credited to Harry Markowitz, whose 1950s work on linear programming pioneered them. In AI it gained prominence through sparse coding (Bruno Olshausen and David Field, 1996), the lasso (Robert Tibshirani, 1996), network pruning, and sparsely gated mixture-of-experts layers (Noam Shazeer and colleagues, 2017).
Usage note
'Sparse' and 'dense' also describe whole models: a dense model uses all its parameters for every input, a sparse one does not. In search, sparse retrievalsparse retrieval, n.: Finding documents by the words they contain, giving more weight to rare, telling words — the classic way search engines work. means keyword-style vectors, a related but distinct usage.
Related
- pruningpruning, n.: Cutting away the parts of an AI model that contribute least, as a gardener trims branches, so the model becomes smaller or faster while still working nearly as well.
- N:M sparsityN:M sparsity, n.: A tidy pattern of zeros inside an AI model, such as two zeros in every group of four numbers, that special chip hardware can skip to run the model faster.
- mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
- active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests.
- sparse attentionsparse attention, n.: Attention in which each word looks at only a selected subset of the other words instead of all of them, so very long texts can be processed at a fraction of the cost.
- sparse autoencodersparse autoencoder, n.: A small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
- model compressionmodel compression, n.: The family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy.
- sparse retrievalsparse retrieval, n.: Finding documents by the words they contain, giving more weight to rare, telling words — the classic way search engines work.
Mentioned in
active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests., FLOPSFLOPS, abbr.: Floating-point operations per second: how many arithmetic steps a computer can do each second. It is the standard speed rating for supercomputers and AI chips., lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model., model compressionmodel compression, n.: The family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy., pruningpruning, n.: Cutting away the parts of an AI model that contribute least, as a gardener trims branches, so the model becomes smaller or faster while still working nearly as well., tensor coretensor core, n.: A small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can., Huang's lawHuang's law, n.: A claim, named after Nvidia's chief executive, that AI chip performance improves much faster than Moore's law because chips, software and number formats all improve together., lottery ticket hypothesislottery ticket hypothesis, n.: The idea that a large, randomly started neural network contains a much smaller network inside it that, trained alone from the same starting values, would learn just as well — a winning lottery ticket., N:M sparsityN:M sparsity, n.: A tidy pattern of zeros inside an AI model, such as two zeros in every group of four numbers, that special chip hardware can skip to run the model faster., sparse attentionsparse attention, n.: Attention in which each word looks at only a selected subset of the other words instead of all of them, so very long texts can be processed at a fraction of the cost., TOPSTOPS, abbr.: Trillions of operations per second: the speed rating usually given for AI chips in phones and laptops, often counting simple whole-number math rather than precise fractions.