chain rule

Calculus

The rule for differentiating a composition of functions: the derivative of f(g(x)) is f′(g(x)) · g′(x), and for functions of several variables the JacobiansJacobian, n.: A table listing how each output of a process responds to a small change in each input. It describes how a transformation locally stretches, squeezes and turns space. of the stages multiply. A neural network is a long composition of simple functions, so the partial derivativespartial derivative, n.: How much a quantity changes when you nudge just one of its many inputs while holding all the others fixed. of its lossloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down. with respect to each weight are products of many local derivatives. Backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction. and reverse-mode automatic differentiationautomatic differentiation, n.: A technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically. apply the chain rule systematically from the output backward, reusing shared intermediate products, so that all the gradients together cost only a small constant multiple of a forward pass (typically about twice its arithmetic). Long products of factors smaller or larger than one are the root of vanishingvanishing gradient problem, n.: A training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn. and exploding gradientsexploding gradient problem, n.: A training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down..

The custom kernel broke the chain rule because its backward function ignored the second input.

Probability

The identity that factors a joint probability into a product of conditionals: p(x₁, …, xₙ) = p(x₁) p(x₂ | x₁) ⋯ p(xₙ | x₁, …, xₙ₋₁), valid for any ordering of the variables. It underlies autoregressiveautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back. language models, which score a whole text by multiplying the conditional probabilitiesconditional probability, n.: The chance of something once you already know that something else is true — for example, the chance of rain given that the sky is cloudy. of each tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. given the tokens before it, and generate text by samplingsampling, n.: Picking some items at random from a larger set or from a range of possibilities — surveying 1,000 voters, say, or letting an AI choose its next word by a weighted lottery. one token at a time.