Skip to content
Last Dictionary
Plane
Language Models & TransformersMathematics & StatisticsAI Engineering & MLOps

log probability

nounAlso: logprob, logprobs, log-prob, token log probability, top logprobs

In plain wordsThe logarithm of the probability a model gives to a token or a text. Computers use it because multiplying many tiny probabilities runs into rounding problems, whereas adding their logarithms does not.

  1. The natural logarithm of the probability a model assigns to an outcome — for a language model, the probability of a tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. given the preceding context, log p(x_t | x_<t). It is always ≤ 0, with 0 meaning certainty. The log probability of a sequence is the sum of its tokens' log probabilities, which avoids the numerical underflow of multiplying many small numbers. Models compute it by applying log-softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score. to the logitslogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity.; the negative mean log probability is the training loss (cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.) and its exponential the perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll.. Many model APIs can return per-token logprobs and the most likely alternatives at each position, which developers use for confidence estimates, calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., classification by comparing candidate answers, and flagging uncertain spans; multiple-choice scoring in many benchmarksbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. relies on them.

    Request the top five logprobs and flag any answer whose first token falls below −1.5.

log⁡pθ(x1:T)=∑t=1Tlog⁡pθ(xt∣x<t)\log p_\theta(x_{1:T}) = \sum_{t=1}^{T} \log p_\theta(x_t \mid x_{<t})

Origin

Log probabilities and log-likelihoods are long-standing statistical tools. The shorthand 'logprobs' entered everyday developer usage with OpenAI's completions API (2020), which could return per-token log probabilities; other APIs and serving engines followed.

Usage note

Availability varies: some APIs omit logprobs for some models, including certain reasoning models. Probabilities of individual tokens are not the same as the probability that an answer is correct.

Related

See also
  • perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll.
  • cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
  • logitlogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity.
  • softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.
  • log-likelihoodlog-likelihood, n.: The logarithm of how probable a model finds the observed data. Taking logs turns a product of many tiny probabilities into a manageable sum; training usually minimizes its negative.
  • calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.

Mentioned in

beam searchbeam search, n.: A search method that keeps several of the best partial answers at each step, rather than only the single best, to find a better overall result without trying every possibility., evaluation harnessevaluation harness, n.: Software that puts AI models through tests in a fixed, repeatable way — feeding in the questions, collecting the answers and scoring them — so that every model is judged identically., logitlogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity., log-likelihoodlog-likelihood, n.: The logarithm of how probable a model finds the observed data. Taking logs turns a product of many tiny probabilities into a manageable sum; training usually minimizes its negative., OpenAI-compatible APIOpenAI-compatible API, n.: An AI service that accepts the same request format as OpenAI's, so software written for OpenAI's models can use other companies' or self-hosted models by changing an address and key., perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll., uncertainty quantificationuncertainty quantification, n.: Measuring how unsure an AI system's predictions are, and why, so that people know when to trust an answer and when to double-check it.