Skip to content
Last Dictionary
Plane
Mathematics & StatisticsDeep Learning

Taylor series

nounAlso: Taylor expansion, Taylor's theorem, Taylor polynomial, Maclaurin series, first-order approximation, second-order approximation

In plain wordsA way of approximating a complicated curve near a chosen point by adding up simple pieces: a constant, a straight-line slope, a bend, and so on.

  1. The expansion of a smooth function around a point a as a power series in (x − a) whose coefficients come from the function's derivatives at a: f(a), plus f′(a)(x − a), plus f″(a)(x − a)²/2!, and so on; for analytic functions the series converges to the function near a. Truncating after a few terms gives a Taylor polynomial whose error Taylor's theorem bounds. Truncated expansions are the reasoning behind most optimization: gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down. minimizes a first-order (linear) approximation of the lossloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down. within a small step; Newton-type second-order optimizationsecond-order optimization, n.: Training methods that use not only the slope of the error but also how that slope is changing — its curvature — to choose better-sized steps. They can need far fewer steps, but each costs more. minimizes a quadratic one built from the HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass.; and XGBoostXGBoost, prop. n.: A fast, free software tool that makes predictions by adding up hundreds of small decision trees, each one fixing the mistakes of those before it. For years it was behind many winning entries in competitions on spreadsheet-style data. fits each tree to a second-order expansion of the loss. Taylor arguments also justify learning ratelearning rate, n.: How big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls. bounds, the linearized regime of the neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior. and attribution methods that linearize a network.

    Expand the loss to second order around the current weights and the Newton step falls out.

f(x)=∑k=0∞f(k)(a)k! (x−a)kf(x) = \sum_{k=0}^{\infty} \frac{f^{(k)}(a)}{k!}\,(x-a)^k

Origin

Named after Brooke Taylor, who published the series in Methodus incrementorum directa et inversa (1715); James Gregory, Isaac Newton, Leibniz and Johann Bernoulli knew versions earlier. Colin Maclaurin's Treatise of Fluxions (1742) credited Taylor, and the special case at zero now bears Maclaurin's name.

Related

See also
  • derivativederivative, n.: How fast something changes as one of its inputs changes — the steepness of a curve at a point. Training an AI means repeatedly following such slopes downhill.
  • HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass.
  • gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.
  • second-order optimizationsecond-order optimization, n.: Training methods that use not only the slope of the error but also how that slope is changing — its curvature — to choose better-sized steps. They can need far fewer steps, but each costs more.
  • neural tangent kernelneural tangent kernel, n.: A mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior.
  • Fourier transformFourier transform, n.: A way of breaking a signal — a sound, an image, any pattern over time or space — into the simple waves of different frequencies that add up to it, much as a prism splits white light into colors.

Mentioned in

Fourier transformFourier transform, n.: A way of breaking a signal — a sound, an image, any pattern over time or space — into the simple waves of different frequencies that add up to it, much as a prism splits white light into colors.