Huber loss
In plain wordsA way of measuring prediction errors that treats small errors like squared distance but large ones more gently, so a few wild data points cannot drag the whole model off course.
A loss functionloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down. for regressionregression, n.: Predicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known. that penalizes residuals smaller than a threshold δ in proportion to their square and larger ones in proportion to their size, the two pieces meeting without a kink. It behaves like mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better. near zero — smooth, with well-behaved gradients — and like mean absolute errormean absolute error, n.: A way of scoring numerical predictions: average how far off each prediction is, ignoring direction, so the score reads as 'typically off by this much'. Lower is better. for large residuals, limiting the influence of outliers. It is the canonical M-estimator of robust statistics; a threshold of about 1.345 times the noise scale keeps 95% of least squares' efficiency when errors are Gaussian. In deep learning the δ = 1 version, called smooth L1 loss, is standard for bounding-box regression in object detectors, and the same idea bounds temporal-difference errors in deep Q-networksdeep Q-network, n.: A method, made famous by DeepMind in 2013–2015, in which a neural network learned to play dozens of Atari video games from screen pixels and the score alone, many at human level.; gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it. libraries offer it as an objective. The pseudo-Huber loss, δ²(√(1 + (a/δ)²) − 1) for a residual a, approximates it with a curve that is smooth to every order.
A few sensor glitches dominated the squared error, so we switched to Huber loss with delta set to two degrees.
Origin
Introduced by Peter J. Huber in 'Robust Estimation of a Location Parameter' (Annals of Mathematical Statistics, 1964), a founding paper of robust statistics. Ross Girshick's Fast R-CNN detector (2015) popularized the δ = 1 'smooth L1' form in computer vision.
Related
- loss functionloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down.
- mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better.
- mean absolute errormean absolute error, n.: A way of scoring numerical predictions: average how far off each prediction is, ignoring direction, so the score reads as 'typically off by this much'. Lower is better.
- regressionregression, n.: Predicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known.
- quantile regressionquantile regression, n.: Predicting not just a typical value but a range — say, the delivery time that 90% of orders beat — by fitting models for chosen percentiles of the outcome instead of its average.
- deep Q-networkdeep Q-network, n.: A method, made famous by DeepMind in 2013–2015, in which a neural network learned to play dozens of Atari video games from screen pixels and the score alone, many at human level.
- mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better.
Sources
- Huber, P. J. (1964). Robust Estimation of a Location Parameter. The Annals of Mathematical Statistics, 35(1), 73–101.