anomaly detection
In plain wordsAutomatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network.
The task of picking out the few cases that stand apart from the bulk of a dataset or data stream, such as a transaction, sensor reading or log event unlike anything normally seen. What can be done depends on the labels available: most often there are none, and an unsupervised method ranks points by how unusual they look; in novelty (semi-supervised) detection a model learns what normal looks like from clean examples and flags departures; classifiers trained on labeled anomalies are used less, because anomalies are scarce and new kinds may not resemble those seen before. Methods include statistical tests and density estimationdensity estimation, n.: Working out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare. (low-likelihood points are suspect), distance- and density-based scores such as nearest-neighbor distance, the local outlier factor and DBSCANDBSCAN, n.: A clustering method that finds groups as crowded regions of data separated by sparse ones, so it can discover oddly shaped groups and set stray points aside as noise. noise points, isolation forestsisolation forest, n.: A method for spotting unusual data points by repeatedly slicing the data at random: odd points get cut off from the rest after only a few slices, while ordinary points take many., one-class support vector machinessupport vector machine, n.: A classic method that separates two groups of examples with the widest possible gap — like laying the broadest possible road between them. A mathematical shortcut lets it draw curved boundaries too., and reconstruction error from autoencodersautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data. or PCAprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.. Anomalies may be point anomalies, contextual (unusual only in context, such as time of day) or collective. Applications include fraud, intrusion detection, industrial monitoring, medicine and model monitoringmodel monitoring, n.: Continuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm..
Anomaly detection on the login telemetry flagged a burst of sign-ins from an unfamiliar region at 3 a.m.
Origin
Statisticians have debated how to treat outlying observations since the 19th century; Douglas Hawkins's 1980 book Identification of Outliers supplied the much-quoted 'different mechanism' definition. Dorothy Denning's intrusion-detection model (presented 1986, published 1987) brought anomaly detection into computer security, and the data-mining boom of the late 1990s and 2000s added scalable density- and isolation-based algorithms.
Usage note
Evaluation is tricky: when anomalies make up well under 1% of data, accuracyaccuracy, n.: The share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others. is uninformative, so precision–recall measures are preferred. 'Outlier' can mean a data error to be removed or the very signal being sought.
Related
- isolation forestisolation forest, n.: A method for spotting unusual data points by repeatedly slicing the data at random: odd points get cut off from the rest after only a few slices, while ordinary points take many.
- density estimationdensity estimation, n.: Working out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare.
- DBSCANDBSCAN, n.: A clustering method that finds groups as crowded regions of data separated by sparse ones, so it can discover oddly shaped groups and set stray points aside as noise.
- autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data.
- class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters.
- model monitoringmodel monitoring, n.: Continuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm.
- out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable.
Sources
- Hawkins, D. M. (1980). Identification of Outliers. London: Chapman and Hall.
- Denning, D. E. (1987). An Intrusion-Detection Model. IEEE Transactions on Software Engineering SE-13(2).
- Chandola, V., Banerjee, A. & Kumar, V. (2009). Anomaly Detection: A Survey. ACM Computing Surveys 41(3).
Mentioned in
object detectionobject detection, n.: Finding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene., autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data., class imbalanceclass imbalance, n.: When one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters., data cleaningdata cleaning, n.: Finding and fixing errors in a dataset — typos, duplicates, missing values, impossible numbers, wrong labels — before using it, because flawed data produces flawed results., data miningdata mining, n.: Searching large collections of data with computers to discover useful patterns, such as products often bought together or signs that a transaction may be fraudulent., DBSCANDBSCAN, n.: A clustering method that finds groups as crowded regions of data separated by sparse ones, so it can discover oddly shaped groups and set stray points aside as noise., density estimationdensity estimation, n.: Working out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare., isolation forestisolation forest, n.: A method for spotting unusual data points by repeatedly slicing the data at random: odd points get cut off from the rest after only a few slices, while ordinary points take many., out-of-distributionout-of-distribution, adj.: Describes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable., unsupervised learningunsupervised learning, n.: Machine learning without answer keys: the computer looks at unlabeled data and finds structure on its own, such as groups of similar customers or the main ways in which data vary., mechanistic anomaly detectionmechanistic anomaly detection, n.: Flagging cases where an AI reaches its output by an unusual internal route — for instance because it is being tricked, or is tricking us — even when the output itself looks normal.