Skip to content
Last Dictionary
Plane
Machine LearningEvaluation & BenchmarksMathematics & Statistics

log loss

nounAlso: logarithmic loss, logistic loss, negative log-likelihood, NLL, logarithmic scoring rule, log score

In plain wordsA score for probability predictions that punishes confident mistakes severely: saying '99% sure' and being wrong costs far more than saying '60% sure' and being wrong. Lower is better.

  1. An evaluation metric and training loss for probabilistic classifiers: the average negative logarithm of the probability the model assigned to the true class. For a binary label y (0 or 1) and predicted probability p, each example contributes −[y log p + (1 − y) log(1 − p)]; the multiclass form equals the cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. between one-hot labels and the predicted distribution, and minimizing it is equivalent to maximum-likelihoodmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this. fitting. Log loss is a strictly proper scoring ruleproper scoring rule, n.: A way of grading probability forecasts that gives the best expected score only to a forecaster who reports what they really believe, so there is nothing to gain by exaggerating or hedging., minimized in expectation only by reporting true probabilities, so it rewards both discrimination and calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.. It is unbounded: a near-zero probability on the true class incurs a huge penalty, so predictions are usually clipped away from 0 and 1. A useful reference point is the log loss of always predicting the base rate. In language modeling the same quantity per token, the negative log-likelihood, is the training objective, and its exponential is perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll..

    The ensemble matched the single model's accuracy but had much better log loss, because it stopped being overconfident.

L=−1N∑i=1N[ yilog⁡pi+(1−yi)log⁡(1−pi) ]\mathcal{L} = -\frac{1}{N} \sum_{i=1}^{N} \big[\, y_i \log p_i + (1 - y_i) \log (1 - p_i) \,\big]

Origin

The logarithmic scoring rule for probability forecasts was proposed by the statistician I. J. Good in 'Rational Decisions' (1952). The short name 'log loss' became common in machine learning in the 2000s and 2010s, notably as an evaluation metric in Kaggle competitions.

Related

See also
  • cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
  • Brier scoreBrier score, n.: A score for probability forecasts: the average squared gap between each forecast probability and what actually happened (1 if it did, 0 if not). Zero is perfect; lower is better.
  • proper scoring ruleproper scoring rule, n.: A way of grading probability forecasts that gives the best expected score only to a forecaster who reports what they really believe, so there is nothing to gain by exaggerating or hedging.
  • calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
  • logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve.
  • perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll.
  • maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this.

Sources

  1. Good, I. J. (1952). Rational Decisions. Journal of the Royal Statistical Society, Series B 14(1).
  2. Gneiting, T. & Raftery, A. E. (2007). Strictly Proper Scoring Rules, Prediction, and Estimation. Journal of the American Statistical Association 102(477).

Mentioned in

Brier scoreBrier score, n.: A score for probability forecasts: the average squared gap between each forecast probability and what actually happened (1 if it did, 0 if not). Zero is perfect; lower is better., calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., click-through rate predictionclick-through rate prediction, n.: Estimating how likely a particular person is to click on a particular ad, search result or recommended item: a prediction made billions of times a day to decide what each user is shown., cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text., empirical risk minimizationempirical risk minimization, n.: The basic recipe of most machine learning: from the models on offer, pick the one with the smallest errors on the training examples, trusting that score as a guide to how it will do on new data., gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it., logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve., log-likelihoodlog-likelihood, n.: The logarithm of how probable a model finds the observed data. Taking logs turns a product of many tiny probabilities into a manageable sum; training usually minimizes its negative., maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this., expected calibration errorexpected calibration error, n.: A single number measuring how far a model's confidence is from its actual hit rate, averaged across confidence levels; zero means its confidence can be taken at face value., hinge losshinge loss, n.: A way of scoring a classifier's mistakes that charges nothing for answers that are right with room to spare, and charges more and more the further an answer falls on the wrong side., proper scoring ruleproper scoring rule, n.: A way of grading probability forecasts that gives the best expected score only to a forecaster who reports what they really believe, so there is nothing to gain by exaggerating or hedging.