derivative

The limit, as a step h shrinks to zero, of the difference quotient [f(x + h) − f(x)]/h: the rate at which a function's output responds to its input at one point, written f′(x) or df/dx. It supplies the local linear model f(x + h) ≈ f(x) + f′(x)h, whose graph is the tangent line, so the derivative is also the curve's slope there. Neural networks are built almost entirely from differentiable operations so that the derivative of the lossloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down. with respect to every parameterparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned. can be computed by automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically. and backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction. and then used by gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.. Functions of many inputs have partial derivativespartial derivative, n.: How much a quantity changes when you nudge just one of its many inputs while holding all the others fixed., collected in the gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill.; second derivatives measure curvature (see HessianHessian, n.: A table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass.). Points where the derivative is zero are stationary points: minima, maxima or saddle pointssaddle point, n.: A spot on a surface that is a low point in one direction but a high point in another, like the middle of a horse's saddle or a mountain pass..

ReLU has no derivative at exactly zero, so frameworks simply use zero there.