Skip to content
Last Dictionary
Plane
Language Models & TransformersEvaluation & BenchmarksMathematics & Statistics

perplexity

nounAlso: PPL, bits per byte, bits per character, BPB

In plain wordsA score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll.

  1. The exponential of the average negative log probabilitylog probability, n.: The logarithm of the probability a model gives to a token or a text. Computers use it because multiplying many tiny probabilities runs into rounding problems, whereas adding their logarithms does not. a language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own. assigns to each token of a held-out text — equivalently, the exponentiated cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. between the data and the model. It can be read as an effective branching factor: a model with perplexity k is, on average, as unsure of each next token as one would be about the roll of a fair k-sided die. Lower is better, and minimizing the usual training loss directly minimizes it. Because it is measured per token, perplexity is comparable only between models with the same tokenizertokenizer, n.: The component that splits text into tokens and converts them to numbers before an AI model sees it, then turns the model's output numbers back into text. and test set; tokenizer-independent variants normalize by characters or bytes (bits per byte). It was the main language-model metric through the 2010s and still tracks pretraining progress, but correlates imperfectly with usefulness on downstream tasks, and suspiciously low perplexity on a benchmark can signal data contaminationdata contamination, n.: When the test questions used to grade an AI system were already in its training data, so a high score may reflect memorized answers rather than real ability..

    Validation perplexity dropped from 9.1 to 8.4 after we cleaned the code data.

PPL(x1:N)=exp⁡ ⁣(−1N∑i=1Nlog⁡pθ(xi∣x<i))\mathrm{PPL}(x_{1:N}) = \exp\!\left( -\frac{1}{N} \sum_{i=1}^{N} \log p_\theta(x_i \mid x_{<i}) \right)

Origin

Introduced as a measure of the difficulty of speech-recognition tasks by Frederick Jelinek, Robert Mercer, Lalit Bahl and James Baker at IBM in 1977, drawing on Claude Shannon's information theory.

Usage note

Not to be confused with Perplexity AI, an AI search company of the same name.

Related

See also
  • log probabilitylog probability, n.: The logarithm of the probability a model gives to a token or a text. Computers use it because multiplying many tiny probabilities runs into rounding problems, whereas adding their logarithms does not.
  • cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
  • language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own.
  • next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers.
  • n-gram modeln-gram model, n.: An early type of language model that predicts the next word from just the few words before it, using counts of how often each word followed those words in a large collection of text.
  • loss functionloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down.

Sources

  1. Jelinek, F., Mercer, R. L., Bahl, L. R. & Baker, J. K. (1977). Perplexity — a measure of the difficulty of speech recognition tasks. Journal of the Acoustical Society of America 62(S1): S63. doi.org/10.1121/1.2016299

Mentioned in

language modellanguage model, n.: A system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own., AI detectorAI detector, n.: Software that scores how likely it is that an AI, not a person, produced some text or media; for writing, its verdicts are wrong often enough that many schools no longer trust them., cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text., information theoryinformation theory, n.: A branch of mathematics that treats information as surprise — the less predictable a message, the more it tells you — and works out how far data can be compressed and how reliably it can be sent., log losslog loss, n.: A score for probability predictions that punishes confident mistakes severely: saying '99% sure' and being wrong costs far more than saying '60% sure' and being wrong. Lower is better., log probabilitylog probability, n.: The logarithm of the probability a model gives to a token or a text. Computers use it because multiplying many tiny probabilities runs into rounding problems, whereas adding their logarithms does not., log-likelihoodlog-likelihood, n.: The logarithm of how probable a model finds the observed data. Taking logs turns a product of many tiny probabilities into a manageable sum; training usually minimizes its negative., model compressionmodel compression, n.: The family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy., n-gram modeln-gram model, n.: An early type of language model that predicts the next word from just the few words before it, using counts of how often each word followed those words in a large collection of text., Penn TreebankPenn Treebank, prop. n.: A collection of English text, best known for its million words of newspaper articles, in which experts marked each word's grammatical role and drew each sentence's structure as a tree; it trained and tested a generation of language software., Perplexity AIPerplexity AI, prop. n.: An American company whose AI search engine answers questions in full sentences with links to its sources; news publishers have sued it for using their articles., ALiBiALiBi, abbr.: A simple way to give a transformer a sense of word order: attention between two words is penalized more the farther apart they are, which helps it handle texts longer than those it trained on., attention sinkattention sink, n.: A quirk of language models: they pour a large share of their attention onto the very first word of the text, even when it means nothing, as a place to park attention they do not need., Frederick JelinekFrederick Jelinek, prop. n.: A Czech-born American engineer who led IBM's speech-recognition research and made statistics learned from data, rather than hand-written grammar rules, the basis of language technology., GPTQGPTQ, prop. n.: A popular method for shrinking large language models to about 4 bits per number after training, carefully adjusting the remaining numbers to make up for each rounding error., label smoothinglabel smoothing, n.: A training trick that tells a model the correct answer is very likely rather than absolutely certain, which stops it from becoming overconfident and often helps it generalize., SwiGLUSwiGLU, n.: A design for the feed-forward layers inside most modern language models: one calculation produces values, another produces a smooth on/off switch, and the two are multiplied together.