Skip to content
Last Dictionary
Plane
Deep LearningMachine Learning

label smoothing

nounAlso: label-smoothing regularization, LSR, soft labels

In plain wordsA training trick that tells a model the correct answer is very likely rather than absolutely certain, which stops it from becoming overconfident and often helps it generalize.

  1. A regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data. technique for classifiers trained with cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. that replaces each one-hot target with a mixture of the one-hot vector and a uniform distribution: the correct class receives probability 1 − ε and the remaining ε is spread evenly over all K classes (ε = 0.1 is typical). Because a softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score. can only approach a one-hot output as logitlogit, n.: A raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity. gaps grow without bound, hard targets keep pushing logits apart; smoothed targets cap the optimal gap, reducing overconfidence and often improving accuracy and calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.. The original transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it. used it, accepting worse perplexityperplexity, n.: A score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll. for better BLEU scores. Drawbacks: it tends to erase information about similarities between wrong classes from the network's representations, which can make the network a worse teacher for knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run., and it fits poorly when labels are already uncertain or when well-calibrated likelihoods are the goal.

ykLS=(1−ε) yk+εKy_k^{\mathrm{LS}} = (1-\varepsilon)\, y_k + \frac{\varepsilon}{K}

Origin

Introduced as 'label-smoothing regularization' by Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jonathon Shlens and Zbigniew Wojna in 'Rethinking the Inception Architecture for Computer Vision' (arXiv December 2015; CVPR 2016). Rafael Müller, Simon Kornblith and Geoffrey Hinton analyzed its effects in 'When Does Label Smoothing Help?' (2019).

Related

See also
  • cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
  • softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.
  • calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
  • regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
  • knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.
  • data augmentationdata augmentation, n.: Creating extra training examples by changing existing ones in ways that keep their meaning — a flipped, cropped or recolored photo of a cat still shows a cat — so the model learns more robustly.

Sources

  1. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J. & Wojna, Z. (2016). Rethinking the Inception Architecture for Computer Vision. CVPR 2016. arxiv.org/abs/1512.00567
  2. Müller, R., Kornblith, S. & Hinton, G. (2019). When Does Label Smoothing Help? NeurIPS 2019.

Mentioned in

data augmentationdata augmentation, n.: Creating extra training examples by changing existing ones in ways that keep their meaning — a flipped, cropped or recolored photo of a cat still shows a cat — so the model learns more robustly., focal lossfocal loss, n.: A way of scoring a model's mistakes during training that pays less attention to examples it already gets right easily, so that learning concentrates on the hard and rare cases., regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.