Skip to content
Last Dictionary
Plane
Data & DatasetsMachine LearningMathematics & Statistics

label shift

nounAlso: prior probability shift, prior shift, target shift

In plain wordsA kind of mismatch where the mix of answers changes — say, a disease becomes far more common during an outbreak — while each individual case still looks the same as before.

  1. A form of distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested. in which the marginal distribution of labels changes between training and deployment, p_train(y) ≠ p_test(y), while the class-conditional distribution of inputs, p(x | y), stays fixed — the natural assumption when causes (diseases, species) produce effects (symptoms, images). A classifier trained under one set of class proportions then gives miscalibrated probabilities and suboptimal decisions under another. Because p(x | y) is unchanged, the new class proportions can be estimated from unlabeled deployment data: classically with an expectation–maximization procedure, or with Black Box Shift Estimation, which inverts a classifier's confusion matrixconfusion matrix, n.: A table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up. measured on held-out data. The model's output probabilities are then reweighted by the ratio of new to old class priors and renormalized.

    Flu season brought a label shift: positives jumped from 5% to 30% of cases, so we re-estimated the priors.

ptest(y∣x)∝ptest(y)ptrain(y) ptrain(y∣x)p_{\text{test}}(y \mid x) \propto \frac{p_{\text{test}}(y)}{p_{\text{train}}(y)}\, p_{\text{train}}(y \mid x)

Origin

Prior-probability adjustment of classifier outputs was described by Marco Saerens, Patrice Latinne and Christine Decaestecker (Neural Computation, 2002); the name 'label shift' was popularized by Zachary Lipton, Yu-Xiang Wang and Alexander Smola's 'Detecting and Correcting for Label Shift with Black Box Predictors' (2018).

Usage note

Distinguished from covariate shiftcovariate shift, n.: A kind of mismatch where the inputs an AI system sees change — say, photos from a new camera — while the right answer for any given input stays the same., where inputs change but p(y | x) is stable. Real shifts are often mixtures, and checking which assumption holds matters more than the choice of correction.

Related

See also
  • distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.
  • class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters.
  • calibrationcalibration, n.: How well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
  • confusion matrixconfusion matrix, n.: A table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up.
  • out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable.
Contrast with
  • covariate shiftcovariate shift, n.: A kind of mismatch where the inputs an AI system sees change — say, photos from a new camera — while the right answer for any given input stays the same.

Sources

  1. Lipton, Z. C., Wang, Y.-X. & Smola, A. (2018). Detecting and Correcting for Label Shift with Black Box Predictors. ICML 2018. arXiv:1802.03916. arxiv.org/abs/1802.03916
  2. Saerens, M., Latinne, P. & Decaestecker, C. (2002). Adjusting the Outputs of a Classifier to New a Priori Probabilities: A Simple Procedure. Neural Computation 14(1).

Mentioned in

concept driftconcept drift, n.: When the real-world link between what a model sees and the right answer changes over time — as when spammers change tactics — so lessons the model learned no longer hold., distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested., covariate shiftcovariate shift, n.: A kind of mismatch where the inputs an AI system sees change — say, photos from a new camera — while the right answer for any given input stays the same.