linear regression
In plain wordsFitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest.
A regressionregression, n.: Predicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known. model in which the predicted response is a linear function of the input featuresfeature, n.: One piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself. — one weight per feature plus an intercept — with any remaining variation treated as noise. 'Linear' means linear in the parameters, so transformed inputs such as x², log x or interaction terms are allowed; polynomial regressionpolynomial regression, n.: Fitting a curve instead of a straight line by also using the square, cube and higher powers of an input. Higher powers let the curve bend more but easily start chasing noise. is a special case. Parameters are usually estimated by ordinary least squaresordinary least squares, n.: The standard way to fit a line through data: choose the line that makes the total of the squared vertical distances from the points to the line as small as possible., which minimizes the sum of squared residuals and has a closed-form solution; under Gaussian noise this coincides with maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this.. With many or correlated features, penalized variants — ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs., the lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model. and the elastic netelastic net, n.: A blend of two ways of keeping a model simple: it can drop useless inputs entirely while treating groups of similar inputs alike, instead of picking one of them at random. — accept a little bias for much lower variance. Simple linear regression has one input; multiple linear regression has several. Each coefficient is interpretable as the expected change in the response per unit change in its feature, holding the others fixed.
A linear regression on floor area alone explains about 60% of the variance in sale prices.
Origin
The least-squares method was published by Adrien-Marie Legendre in 1805 and by Carl Friedrich Gauss in 1809, both to determine the orbits of comets and planets from observations. The name came later, from Francis Galton's 1880s heredity studies; Udny Yule and Karl Pearson developed linear regression as a general statistical tool around 1900.
Usage note
Usually the first baseline worth fitting. Correlated inputs (multicollinearity) make individual coefficients unstable even when predictions stay accurate, and coefficients describe associations, not causal effects, unless the design supports causal inferencecausal inference, n.: Working out whether one thing actually causes another, rather than merely occurring alongside it — for example, whether a new feature made users stay longer or just happened at the same time..
Related
- ordinary least squaresordinary least squares, n.: The standard way to fit a line through data: choose the line that makes the total of the squared vertical distances from the points to the line as small as possible.
- regressionregression, n.: Predicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known.
- polynomial regressionpolynomial regression, n.: Fitting a curve instead of a straight line by also using the square, cube and higher powers of an input. Higher powers let the curve bend more but easily start chasing noise.
- ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs.
- lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model.
- generalized linear modelgeneralized linear model, n.: A family of statistical models that extends straight-line regression to outcomes such as yes/no answers, counts or waiting times, by linking the average outcome to a weighted sum of inputs through a suitable curve.
- logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve.
- logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve.
- nonparametric modelnonparametric model, n.: A model that lets the data decide how complex it becomes: the more examples it gets, the more detail it can capture — for instance, one that predicts by comparing a new case with every stored example.
Sources
- Legendre, A.-M. (1805). Nouvelles méthodes pour la détermination des orbites des comètes. Paris: Firmin Didot.
- Gauss, C. F. (1809). Theoria motus corporum coelestium in sectionibus conicis solem ambientium. Hamburg: Perthes & Besser.
Mentioned in
machine learningmachine learning, n.: A way of building software that learns from examples instead of following rules written by a programmer. Shown thousands of labeled photos, for instance, a program can learn to recognize cats., regressionregression, n.: Predicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known., baggingbagging, n.: Training many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error., coefficient of determinationcoefficient of determination, n.: A score for numerical predictions showing how much of the variation in the true values a model accounts for, compared with always guessing the average: 1 is perfect, 0 is no better than the average., convex optimizationconvex optimization, n.: Finding the lowest point of a bowl-shaped landscape. Because a bowl has only one bottom, such problems can be solved reliably — unlike training a neural network, whose landscape is bumpy., generalized additive modelgeneralized additive model, n.: A prediction model that adds up separate, possibly curved effects of each input, so each input's influence can be drawn as its own graph and read off directly., generalized linear modelgeneralized linear model, n.: A family of statistical models that extends straight-line regression to outcomes such as yes/no answers, counts or waiting times, by linking the average outcome to a weighted sum of inputs through a suitable curve., interpretable modelinterpretable model, n.: A model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed., lassolasso, n.: A version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model., logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve., ordinary least squaresordinary least squares, n.: The standard way to fit a line through data: choose the line that makes the total of the squared vertical distances from the points to the line as small as possible., parameterparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned., parametric modelparametric model, n.: A model whose shape is fixed in advance by a set number of adjustable settings, like a straight line's slope and intercept. More data refines those settings but never adds new ones., polynomial regressionpolynomial regression, n.: Fitting a curve instead of a straight line by also using the square, cube and higher powers of an input. Higher powers let the curve bend more but easily start chasing noise., ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs., scikit-learnscikit-learn, prop. n.: A free, widely taught Python library of classic machine-learning methods — such as decision trees, regression and clustering — that all work in the same simple way., supervised learningsupervised learning, n.: Teaching a computer by example: you give it many inputs paired with the right answers, such as photos labeled 'cat' or 'dog', and it learns to answer correctly for new inputs., quantile regressionquantile regression, n.: Predicting not just a typical value but a range — say, the delivery time that 90% of orders beat — by fitting models for chosen percentiles of the outcome instead of its average.