Skip to content
Last Dictionary
Plane
Machine LearningMathematics & Statistics

ordinary least squares

nounAlso: OLS, least squares, method of least squares, linear least squares, normal equations

In plain wordsThe standard way to fit a line through data: choose the line that makes the total of the squared vertical distances from the points to the line as small as possible.

  1. The method of estimating the parameters of a linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest. model by minimizing the sum of squared residuals, the differences between observed and fitted values. Setting the gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill. to zero gives the normal equations XᵀXβ = Xᵀy, whose solution is unique when the columns of the design matrix X are linearly independent; in practice it is computed by QR decomposition or singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data. rather than by inverting XᵀX, which is numerically fragile. By the Gauss–Markov theorem, OLS is the best linear unbiased estimator when errors have zero mean, constant variance and no correlation; with Gaussian errors it is also the maximum likelihoodmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this. estimate. Squaring makes it sensitive to outliers, and nearly collinear features inflate the variance of its estimates — motivations for robust regression and ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs..

    OLS gave a negative coefficient for advertising spend, which turned out to be collinearity with the seasonal dummies.

β^=arg⁡min⁡β∥y−Xβ∥22=(X⊤X)−1X⊤y\hat{\boldsymbol\beta} = \arg\min_{\boldsymbol\beta} \lVert \mathbf{y} - X\boldsymbol\beta \rVert_2^2 = (X^{\top}X)^{-1}X^{\top}\mathbf{y}

Origin

Adrien-Marie Legendre published the 'méthode des moindres carrés' in 1805 in a work on comet orbits. Carl Friedrich Gauss published it in 1809, saying he had used it since 1795, and in 1821 proved a version of the optimality result now called the Gauss–Markov theorem; their priority dispute is among the most famous in statistics. 'Ordinary' distinguishes it from weighted and generalized least squares.

Related

See also
  • linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest.
  • regressionregression, n.: Predicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known.
  • maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this.
  • ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs.
  • mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better.
  • singular value decompositionsingular value decomposition, n.: A way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data.

Sources

  1. Legendre, A.-M. (1805). Nouvelles méthodes pour la détermination des orbites des comètes. Paris: Firmin Didot.

Mentioned in

coefficient of determinationcoefficient of determination, n.: A score for numerical predictions showing how much of the variation in the true values a model accounts for, compared with always guessing the average: 1 is perfect, 0 is no better than the average., empirical risk minimizationempirical risk minimization, n.: The basic recipe of most machine learning: from the models on offer, pick the one with the smallest errors on the training examples, trusting that score as a guide to how it will do on new data., linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest., mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better., polynomial regressionpolynomial regression, n.: Fitting a curve instead of a straight line by also using the square, cube and higher powers of an input. Higher powers let the curve bend more but easily start chasing noise.