automatic differentiation
In plain wordsA technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically.
A way of making a program that evaluates a function also produce that function's exact derivatives (up to rounding), by breaking the computation into elementary operations whose derivatives are known and combining them with the chain rulechain rule, n.: A rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer.. It differs from symbolic differentiation, which manipulates formulas and can blow up in size, and from finite differences, which are approximate. Forward mode carries derivatives alongside values from inputs to outputs (efficient for few inputs and many outputs; implementable with dual numbers). Reverse mode records the computation and propagates sensitivities from outputs back to inputs, giving the full gradient of a scalar loss with respect to millions or billions of inputs for a small constant multiple of the cost of evaluating it; applied to neural networks, it is backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.. Frameworks such as PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips., JAXJAX, prop. n.: A free Google library that lets researchers write mathematics in familiar Python and automatically get derivatives and very fast code for graphics chips and Google's AI chips. and TensorFlowTensorFlow, prop. n.: Google's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems. implement AD by recording operations into a computational graphcomputational graph, n.: A map of a calculation drawn as boxes and arrows: each box is a simple operation and each arrow carries a number to the next step. AI software uses it to trace errors backward. or transforming code, freeing researchers from deriving gradients by hand; composing the two modes yields higher-order quantities such as Hessian–vector products.
Origin
Robert Wengert described forward-mode derivative evaluation by program in 1964; Seppo Linnainmaa gave reverse mode in his 1970 master's thesis, and Bert Speelpenning built an early reverse-mode program transformer in 1980. Andreas Griewank's work consolidated the field from the 1980s. Machine learning rediscovered much of it independently, as surveyed by Atılım Güneş Baydin and colleagues (2015; JMLR 2018).
Usage note
'Autograd' names both a 2015 Python library from Harvard and PyTorch's AD engine. AD differentiates the program as executed, so loops and branches pose no problem, but operations with zero or undefined derivatives (rounding, sampling) need workarounds such as the straight-through estimatorstraight-through estimator, n.: A shortcut that lets networks learn even when they contain steps with no useful slope, such as rounding: during learning, the computer simply pretends the step passed its input straight through. or the reparameterization trickreparameterization trick, n.: A trick that rewrites a random draw as fixed numbers plus separate, outside randomness, so that a neural network can still learn through a step that involves chance..
Related
- backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
- computational graphcomputational graph, n.: A map of a calculation drawn as boxes and arrows: each box is a simple operation and each arrow carries a number to the next step. AI software uses it to trace errors backward.
- chain rulechain rule, n.: A rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer.
- JacobianJacobian, n.: A table listing how each output of a process responds to a small change in each input. It describes how a transformation locally stretches, squeezes and turns space.
- PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips.
- JAXJAX, prop. n.: A free Google library that lets researchers write mathematics in familiar Python and automatically get derivatives and very fast code for graphics chips and Google's AI chips.
Sources
- Baydin, A. G., Pearlmutter, B. A., Radul, A. A. & Siskind, J. M. (2018). Automatic differentiation in machine learning: a survey. Journal of Machine Learning Research 18(153), 1–43. arxiv.org/abs/1502.05767
- Wengert, R. E. (1964). A simple automatic derivative evaluation program. Communications of the ACM 7(8), 463–464.
Mentioned in
backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction., backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change., chain rulechain rule, n.: A rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer., computational graphcomputational graph, n.: A map of a calculation drawn as boxes and arrows: each box is a simple operation and each arrow carries a number to the next step. AI software uses it to trace errors backward., derivativederivative, n.: How fast something changes as one of its inputs changes — the steepness of a curve at a point. Training an AI means repeatedly following such slopes downhill., gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill., NumPyNumPy, prop. n.: The basic free Python library for working quickly with large grids of numbers; nearly every data-science and machine-learning tool in Python is built on it or copies its style., PyTorchPyTorch, prop. n.: The most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips., JacobianJacobian, n.: A table listing how each output of a process responds to a small change in each input. It describes how a transformation locally stretches, squeezes and turns space., JAXJAX, prop. n.: A free Google library that lets researchers write mathematics in familiar Python and automatically get derivatives and very fast code for graphics chips and Google's AI chips., MLXMLX, prop. n.: A free Apple software toolkit for running and training AI models efficiently on Mac computers with Apple's own chips, popular for running language models locally on a laptop., Paul WerbosPaul Werbos, prop. n.: An American scientist whose 1974 doctoral thesis set out backpropagation, the method of passing errors backward that now trains nearly all neural networks., physics-informed neural networkphysics-informed neural network, n.: A neural network that is penalized during training whenever its output breaks a known physical law, such as an equation of fluid flow, so it can model a system well even from few measurements., Seppo LinnainmaaSeppo Linnainmaa, prop. n.: A Finnish computer scientist who in 1970 first published the method for computing all of a program's derivatives in one backward pass, the mathematical core of backpropagation., straight-through estimatorstraight-through estimator, n.: A shortcut that lets networks learn even when they contain steps with no useful slope, such as rounding: during learning, the computer simply pretends the step passed its input straight through., TheanoTheano, prop. n.: One of the first popular free software tools for deep learning, built at a Montreal university lab; development ended in 2017, but it shaped the toolkits that replaced it.