polynomial regression
In plain wordsFitting a curve instead of a straight line by also using the square, cube and higher powers of an input. Higher powers let the curve bend more but easily start chasing noise.
Regressionregression, n.: Predicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known. that fits the response with a degree-d polynomial of the input, y ≈ β₀ + β₁x + β₂x² + … + β_d x^d, or with several inputs as a sum of monomials including interaction terms. Because the model is linear in its coefficients, it is a special case of multiple linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest. on expanded features and can be fit by ordinary least squaresordinary least squares, n.: The standard way to fit a line through data: choose the line that makes the total of the squared vertical distances from the points to the line as small as possible.. The degree controls flexibility and is the textbook illustration of the bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.: low degrees underfitunderfitting, n.: When a model is too simple, or trained too little, to capture the real pattern in the data, so it does poorly even on the examples it learned from., while high degrees overfitoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding., oscillate between data points (Runge's phenomenon) and extrapolate badly. Raw powers are strongly correlated, so orthogonal polynomials or regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data. are used in practice; splines, which join low-degree pieces smoothly, usually behave better.
A degree-9 polynomial regression passes through all ten training points and is useless on new ones.
Origin
Joseph Diez Gergonne's 1815 paper on choosing observations for fitting polynomials by least squares is cited as the first design of experiments for polynomial regression; Kirstine Smith's 1918 Biometrika paper on the precision of fitted polynomials is a founding work of optimal design. The fitting method itself rests on Legendre's (1805) and Gauss's (1809) least squares.
Related
- linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest.
- overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
- underfittingunderfitting, n.: When a model is too simple, or trained too little, to capture the real pattern in the data, so it does poorly even on the examples it learned from.
- bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.
- regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
- ordinary least squaresordinary least squares, n.: The standard way to fit a line through data: choose the line that makes the total of the squared vertical distances from the points to the line as small as possible.
Sources
- Gergonne, J. D. (1815). Application de la méthode des moindres quarrés à l'interpolation des suites. Annales de Mathématiques Pures et Appliquées, 6, 242–252. English translation in Historia Mathematica, 1 (1974).
Mentioned in
linear regressionlinear regression, n.: Fitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest.