Skip to content
Last Dictionary
Plane
Machine LearningEvaluation & BenchmarksMathematics & Statistics

Brier score

nounAlso: Brier skill score, quadratic score, probability score

In plain wordsA score for probability forecasts: the average squared gap between each forecast probability and what actually happened (1 if it did, 0 if not). Zero is perfect; lower is better.

  1. A proper scoring ruleproper scoring rule, n.: A way of grading probability forecasts that gives the best expected score only to a forecaster who reports what they really believe, so there is nothing to gain by exaggerating or hedging. that grades probability forecasts by their squared distance from what happened: with each outcome coded 1 if it occurred and 0 if not, it is the mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better. of the forecast probabilities, summed over classes in the multiclass form. For yes/no events it ranges from 0, for forecasts always certain and right, to 1, for forecasts always certain and wrong. Being strictly proper, it is optimized in expectation by honest probabilities. Allan Murphy's 1973 decomposition splits it into reliability (a calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time. term), resolution (how much forecasts depart from the base rate when outcomes differ) and uncertainty (the inherent variability of the outcome). Compared with log losslog loss, n.: A score for probability predictions that punishes confident mistakes severely: saying '99% sure' and being wrong costs far more than saying '60% sure' and being wrong. Lower is better., the Brier score is bounded and penalizes confident errors far less harshly, making it more forgiving but less sensitive to rare events. The Brier skill score expresses improvement over a reference forecast, such as climatology in weather forecasting or the base rate in classification.

    Our rain model's Brier score of 0.11 beat the climatology baseline's 0.19.

BS=1N∑i=1N(pi−oi)2\mathrm{BS} = \frac{1}{N} \sum_{i=1}^{N} \left(p_i - o_i\right)^2

Origin

Proposed by Glenn W. Brier of the U.S. Weather Bureau in 'Verification of Forecasts Expressed in Terms of Probability' (Monthly Weather Review, 1950) to evaluate probabilistic weather forecasts. Allan Murphy's 1973 decomposition made it a diagnostic tool, and it is now used in medicine, forecasting tournaments and studies of machine-learning calibration.

Related

See also
  • log losslog loss, n.: A score for probability predictions that punishes confident mistakes severely: saying '99% sure' and being wrong costs far more than saying '60% sure' and being wrong. Lower is better.
  • proper scoring ruleproper scoring rule, n.: A way of grading probability forecasts that gives the best expected score only to a forecaster who reports what they really believe, so there is nothing to gain by exaggerating or hedging.
  • calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
  • mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better.
  • expected calibration errorexpected calibration error, n.: A single number measuring how far a model's confidence is from its actual hit rate, averaged across confidence levels; zero means its confidence can be taken at face value.
  • baselinebaseline, n.: A simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value.

Sources

  1. Brier, G. W. (1950). Verification of Forecasts Expressed in Terms of Probability. Monthly Weather Review 78(1).
  2. Murphy, A. H. (1973). A New Vector Partition of the Probability Score. Journal of Applied Meteorology 12(4).

Mentioned in

calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time., log losslog loss, n.: A score for probability predictions that punishes confident mistakes severely: saying '99% sure' and being wrong costs far more than saying '60% sure' and being wrong. Lower is better., mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better., expected calibration errorexpected calibration error, n.: A single number measuring how far a model's confidence is from its actual hit rate, averaged across confidence levels; zero means its confidence can be taken at face value., proper scoring ruleproper scoring rule, n.: A way of grading probability forecasts that gives the best expected score only to a forecaster who reports what they really believe, so there is nothing to gain by exaggerating or hedging., survival analysissurvival analysis, n.: Methods for predicting how long until something happens — a patient's relapse, a machine's failure, a customer's cancellation — that properly use the cases where it has not happened yet.