regression
In plain wordsPredicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known.
- 1
Machine learning
A supervised learningsupervised learning, n.: Teaching a computer by example: you give it many inputs paired with the right answers, such as photos labeled 'cat' or 'dog', and it learns to answer correctly for new inputs. task in which the target to be predicted is a continuous quantity (a real number or vector) rather than a discrete class. A regression model maps input featuresfeature, n.: One piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself. to a predicted value and is trained by minimizing a loss functionloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down. that measures the gap between predictions and observed targets — most often mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better., whose minimizer is the conditional mean, or mean absolute errormean absolute error, n.: A way of scoring numerical predictions: average how far off each prediction is, ignoring direction, so the score reads as 'typically off by this much'. Lower is better., whose minimizer is the conditional median. Methods range from linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest. and generalized linear modelsgeneralized linear model, n.: A family of statistical models that extends straight-line regression to outcomes such as yes/no answers, counts or waiting times, by linking the average outcome to a weighted sum of inputs through a suitable curve. through regression trees, gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it. and Gaussian processesGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them. to neural networks. Performance is reported with error metrics or the coefficient of determinationcoefficient of determination, n.: A score for numerical predictions showing how much of the variation in the true values a model accounts for, compared with always guessing the average: 1 is perfect, 0 is no better than the average..
Is this regression or classification — do we predict the delay in minutes or only whether the flight is late?
- 2
Statistics
More broadly, any method for estimating how the expected value, or another property such as a quantile, of a response variable depends on explanatory variables. Statistical usage emphasizes interpreting the fitted coefficients, testing hypotheses about them and quantifying their uncertainty, not only predicting well.
- 3
Engineering and evaluation
A decline in a system's performance or behavior after a change, as when a new model checkpoint scores lower than its predecessor on some benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.. The sense is borrowed from software testing, where a regression is a previously working feature that breaks; regression tests and fixed evaluationevaluation, n.: Testing an AI system to find out what it can and cannot do, how well, and how safely — from exam-style quizzes to hands-on trials and expert reviews. suites are rerun after every change to catch such declines.
The new checkpoint is better at code but shows a regression on multilingual math.
Origin
Francis Galton called the tendency of offspring of extreme parents to lie closer to the average 'reversion' in an 1877 lecture on the seed sizes of sweet peas, and 'regression' in his September 1885 presidential address to the anthropology section of the British Association, on human height; his paper 'Regression towards Mediocrity in Hereditary Stature' followed in 1886. Udny Yule and Karl Pearson generalized the idea, and the name passed from the phenomenon to the fitting method, whose least-squares machinery Adrien-Marie Legendre (1805) and Carl Friedrich Gauss (1809) had published decades earlier.
Usage note
Despite its name, logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve. is mainly used for classification (it models a class probability). Galton's phenomenon, regression toward the mean, is a statistical effect rather than a method; overlooking it leads to misreading before-and-after comparisons.
Related
- linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest.
- classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled.
- supervised learningsupervised learning, n.: Teaching a computer by example: you give it many inputs paired with the right answers, such as photos labeled 'cat' or 'dog', and it learns to answer correctly for new inputs.
- logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve.
- quantile regressionquantile regression, n.: Predicting not just a typical value but a range — say, the delivery time that 90% of orders beat — by fitting models for chosen percentiles of the outcome instead of its average.
- mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better.
- coefficient of determinationcoefficient of determination, n.: A score for numerical predictions showing how much of the variation in the true values a model accounts for, compared with always guessing the average: 1 is perfect, 0 is no better than the average.
- classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled.
Sources
- Galton, F. (1886). Regression towards Mediocrity in Hereditary Stature. Journal of the Anthropological Institute of Great Britain and Ireland, 15, 246–263.
Mentioned in
classificationclassification, n.: Sorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled., ARIMAARIMA, n.: A classic statistical recipe for forecasting a series of numbers over time: remove any steady trend, then predict the next value from recent values and recent prediction errors., baggingbagging, n.: Training many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error., coefficient of determinationcoefficient of determination, n.: A score for numerical predictions showing how much of the variation in the true values a model accounts for, compared with always guessing the average: 1 is perfect, 0 is no better than the average., data miningdata mining, n.: Searching large collections of data with computers to discover useful patterns, such as products often bought together or signs that a transaction may be fraudulent., generalized linear modelgeneralized linear model, n.: A family of statistical models that extends straight-line regression to outcomes such as yes/no answers, counts or waiting times, by linking the average outcome to a weighted sum of inputs through a suitable curve., k-nearest neighborsk-nearest neighbors, n.: Predicting something about a new example by finding the few most similar examples already known and going with their majority answer, or their average., linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest., mean absolute errormean absolute error, n.: A way of scoring numerical predictions: average how far off each prediction is, ignoring direction, so the score reads as 'typically off by this much'. Lower is better., mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better., ordinary least squaresordinary least squares, n.: The standard way to fit a line through data: choose the line that makes the total of the squared vertical distances from the points to the line as small as possible., polynomial regressionpolynomial regression, n.: Fitting a curve instead of a straight line by also using the square, cube and higher powers of an input. Higher powers let the curve bend more but easily start chasing noise., predictionprediction, n.: A model's output for a given input: its best guess at an unknown answer. In AI it need not be about the future; labeling a photo 'cat' counts as a prediction., supervised learningsupervised learning, n.: Teaching a computer by example: you give it many inputs paired with the right answers, such as photos labeled 'cat' or 'dog', and it learns to answer correctly for new inputs., time series forecastingtime series forecasting, n.: Predicting future values of something measured over time, such as sales, electricity demand, prices or rainfall, from its past values and related information., CARTCART, abbr.: A classic recipe for building decision trees that splits data two ways at every step, works for both categories and numbers, and then trims the tree back so it does not memorize noise., Huber lossHuber loss, n.: A way of measuring prediction errors that treats small errors like squared distance but large ones more gently, so a few wild data points cannot drag the whole model off course., quantile regressionquantile regression, n.: Predicting not just a typical value but a range — say, the delivery time that 90% of orders beat — by fitting models for chosen percentiles of the outcome instead of its average., survival analysissurvival analysis, n.: Methods for predicting how long until something happens — a patient's relapse, a machine's failure, a customer's cancellation — that properly use the cases where it has not happened yet.