Zipf's law

The empirical regularity that word frequencies in a large body of text fall off roughly as the reciprocal of their rank — the r-th most common word occurs about 1/r as often as the most common one — so that frequency times rank stays roughly constant; more generally, a rank–frequency power lawpower law, n.: A rule linking two quantities in which scaling one up by a given factor always changes the other by the same multiple, wherever you start. Plotted on logarithmic axes, it is a straight line. with exponent near 1. Similar Zipfian distributions describe city sizes, website traffic and citations. For AI the law has practical consequences: most word types, and many tokenstoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. in a large vocabularyvocabulary, n.: The fixed list of all the tokens a language model knows, each with its own number. Everything the model reads or writes must be built from pieces on this list., are rare, which motivates subword tokenizationtokenization, n.: The first step in feeding text to a language model: chopping it into tokens — whole words, word pieces or characters — and turning each into a number the model can work with. such as byte-pair encodingbyte-pair encoding, n.: A popular way to choose the word pieces an AI breaks text into: start from single characters and repeatedly glue together the pair that occurs most often, until the list reaches the desired size.; rare facts occur in training data too seldom to be learned reliably, contributing to failures on the long taillong tail, n.: The huge number of rare things — uncommon words, unusual road situations, obscure facts — that each appear seldom but together make up a large share of what an AI system meets.; frequency imbalance skews embeddingsembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. and calibration; and the heavy-tailed structure of language data has been proposed as one explanation for smooth scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments. and for in-context learningin-context learning, n.: A language model's ability to pick up a new task from instructions or examples placed in its input, on the spot, without any retraining — like a quick study learning a game's rules from one round.. Heaps' law, relating vocabulary size to corpus length, is a companion regularity.

Because token frequencies are Zipfian, many vocabulary entries are seen only rarely during training.