principal component analysis
In plain wordsA technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.
A linear dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from. method that summarizes many correlated variables by a few uncorrelated ones, the principal components: weighted sums of the mean-centered variables, chosen one at a time so that each captures as much of the remaining variance as possible while staying perpendicular to those already found. The weight vectors are eigenvectorseigenvector, n.: A special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies. of the covariance matrix, ranked by their eigenvalues (the variance each component explains), and in practice they are read off a singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data. of the centered data. Keeping the top k components gives the best rank-k reconstruction in squared error (low-rank approximationlow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation.), so PCA is used for compression, denoising, decorrelating features, visualization, and as preprocessing before clusteringclustering, n.: Automatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size. or regression. It captures only linear structure, is sensitive to feature scalefeature scaling, n.: Putting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units. and outliers, and its components can be hard to interpret. Variants include kernel PCA for nonlinear structure, sparse PCA, incremental PCA for streaming data and probabilistic PCA.
The first two principal components explained 71% of the variance, so we plotted the samples on those axes.
Origin
Karl Pearson introduced the method in 1901 in 'On Lines and Planes of Closest Fit to Systems of Points in Space'. Harold Hotelling developed it independently in 1933 and gave it the name 'principal components'. Equivalent techniques arose separately as the Karhunen–Loève transform in signal processing, empirical orthogonal functions in meteorology and proper orthogonal decomposition in engineering.
Usage note
Standardize variables first unless they share units, or the largest-scale variable dominates. PCA ignores labels: the directions of greatest variance need not be those that best separate classes (contrast linear discriminant analysislinear discriminant analysis, n.: A statistical method for telling known groups apart. It finds the weighted mix of measurements on which groups differ most relative to their spread, then uses it to sort new items or compress many measurements into a few.).
Related
- dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from.
- singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data.
- eigenvectoreigenvector, n.: A special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies.
- low-rank approximationlow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation.
- feature scalingfeature scaling, n.: Putting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units.
- latent semantic analysislatent semantic analysis, n.: An early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together.
- autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data.
- linear discriminant analysislinear discriminant analysis, n.: A statistical method for telling known groups apart. It finds the weighted mix of measurements on which groups differ most relative to their spread, then uses it to sort new items or compress many measurements into a few.
Sources
- Pearson, K. (1901). On Lines and Planes of Closest Fit to Systems of Points in Space. Philosophical Magazine 2(11).
- Hotelling, H. (1933). Analysis of a Complex of Statistical Variables into Principal Components. Journal of Educational Psychology 24.
Mentioned in
anomaly detectionanomaly detection, n.: Automatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network., autoencoderautoencoder, n.: A neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data., dimensionality reductiondimensionality reduction, n.: Squeezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from., eigenvectoreigenvector, n.: A special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies., feature extractionfeature extraction, n.: Turning raw data such as pixels or sound into a short list of telling numbers — computed by hand-designed rules or by an AI system trained earlier — that a new model can learn from or compare., feature scalingfeature scaling, n.: Putting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units., Hebbian learningHebbian learning, n.: The idea that a connection between two brain cells grows stronger when they are active at the same time — the principle behind the slogan 'cells that fire together wire together'., kernel trickkernel trick, n.: A mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space., linear discriminant analysislinear discriminant analysis, n.: A statistical method for telling known groups apart. It finds the weighted mix of measurements on which groups differ most relative to their spread, then uses it to sort new items or compress many measurements into a few., low-rank approximationlow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation., matrix factorizationmatrix factorization, n.: Breaking a big table of numbers, such as which users liked which films, into two smaller tables of hidden traits, so missing entries can be filled in by multiplying them back together., singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data., t-SNEt-SNE, n.: A technique for drawing data that have many measurements per item as a flat map in which similar items land close together, widely used to look for groups in AI models' internal data or in genetic data., unsupervised learningunsupervised learning, n.: Machine learning without answer keys: the computer looks at unlabeled data and finds structure on its own, such as groups of similar customers or the main ways in which data vary., activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions., assistant axisassistant axis, n.: The main direction in a chatbot's internal activity that measures how much it is acting as its usual helpful-assistant self, as opposed to some other character it has slipped into., centered kernel alignmentcentered kernel alignment, n.: A score from 0 to 1 measuring how similar two neural networks' internal representations are, by comparing how each arranges the same set of inputs relative to one another., covariancecovariance, n.: A measure of whether two quantities tend to rise and fall together, move in opposite directions, or show no consistent relationship., dictionary learningdictionary learning, n.: A method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts., independent component analysisindependent component analysis, n.: A technique for unmixing signals that have been blended together, such as separating individual voices from several microphones recording a crowded room., orthogonalityorthogonality, n.: The mathematical version of being at right angles. Two directions are orthogonal when moving along one tells you nothing about the other, like north and east on a map., source separationsource separation, n.: Pulling apart a mixture of sounds into its separate parts, such as isolating one voice in a noisy party or the vocals, drums and bass in a finished song.