linear discriminant analysis
In plain wordsA statistical method for telling known groups apart. It finds the weighted mix of measurements on which groups differ most relative to their spread, then uses it to sort new items or compress many measurements into a few.
A method for classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled. and supervised dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from. that models each class as a multivariate normal distribution with its own mean but a covariance matrix shared by all classes. Under these assumptions the Bayes-optimal decision boundariesdecision boundary, n.: The dividing line, or surface, where a classifier switches from predicting one category to another. Simple models draw straight lines; flexible ones draw curves. are linear, and a point is assigned to the class with the highest posterior probability. Equivalently, in Fisher's formulation, LDA finds projections that maximize the ratio of between-class to within-class variance; with K classes there are at most K − 1 such discriminant directions, making LDA a supervised alternative to principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores. for visualizing or compressing labeled data. Allowing each class its own covariance gives quadratic discriminant analysis (QDA), with curved boundaries; shrinking the covariance estimate (regularized LDA) helps when features outnumber observations. LDA is a generative modelgenerative model, n.: A model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them. and a close relative of logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve..
Projecting the spectra onto the two LDA directions separated the three grape varieties cleanly.
Origin
Ronald Fisher introduced the linear discriminant in 'The Use of Multiple Measurements in Taxonomic Problems' (Annals of Eugenics, 1936), illustrated with Edgar Anderson's iris measurements; C. R. Rao generalized it to more than two classes in 1948. Fisher obtained his discriminant by maximizing class separation, without a Gaussian model; the shared-covariance Gaussian formulation is the later probabilistic reading.
Usage note
The abbreviation LDA also stands for latent Dirichlet allocationlatent Dirichlet allocation, n.: A popular topic-finding method that assumes every document is a blend of a few themes and every theme favors certain words, then works backward from the words to find both., an unrelated topic model.
Related
- classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled.
- principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.
- dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from.
- logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve.
- naive Bayesnaive Bayes, n.: A simple, fast way of sorting items into categories using probabilities, which pretends that each clue is independent of the others. It is the classic technique behind early email spam filters.
- decision boundarydecision boundary, n.: The dividing line, or surface, where a classifier switches from predicting one category to another. Simple models draw straight lines; flexible ones draw curves.
- latent Dirichlet allocationlatent Dirichlet allocation, n.: A popular topic-finding method that assumes every document is a blend of a few themes and every theme favors certain words, then works backward from the words to find both.
- principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.
Sources
- Fisher, R. A. (1936). The Use of Multiple Measurements in Taxonomic Problems. Annals of Eugenics, 7(2), 179–188.
- Rao, C. R. (1948). The Utilization of Multiple Measurements in Problems of Biological Classification. Journal of the Royal Statistical Society, Series B, 10(2), 159–203.
Mentioned in
decision boundarydecision boundary, n.: The dividing line, or surface, where a classifier switches from predicting one category to another. Simple models draw straight lines; flexible ones draw curves., dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from., latent Dirichlet allocationlatent Dirichlet allocation, n.: A popular topic-finding method that assumes every document is a blend of a few themes and every theme favors certain words, then works backward from the words to find both., parametric modelparametric model, n.: A model whose shape is fixed in advance by a set number of adjustable settings, like a straight line's slope and intercept. More data refines those settings but never adds new ones., principal component analysisprincipal component analysis, n.: A technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.