variance

The averageexpected value, n.: The long-run average you would get if a chancy situation could be repeated many times — 3.5 for a fair die, even though no single roll shows 3.5., weighted by probability, of the square of a random variablerandom variable, n.: A quantity whose value depends on chance, such as the number rolled on a die or the next word a chatbot will write.'s departure from its own mean, Var(X) = E[(X − E[X])²], which can also be computed as E[X²] − (E[X])²; taking its square root gives the standard deviation σ, a measure of spread in the variable's own units. Variance is zero only for constants, adds across independent variables and scales with the square of any multiplier, so averaging n independent samples divides it by n — the basis of mini-batchmini-batch, n.: A small random handful of training examples that a model learns from in a single step. Training works through many such handfuls instead of studying the whole dataset at once. gradient estimates and of error bars. It shapes much of deep-learning practice: weight initializationweight initialization, n.: Choosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers. schemes choose weight variances that keep activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment. and gradients from exploding or vanishing across layers; batch normalizationbatch normalization, n.: A layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably. and layer normalizationlayer normalization, n.: A layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots. rescale activations to unit variance; AdamAdam, prop. n.: The most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been. divides each update by the square root of a running estimate of the gradient's second moment; and policy-gradient methods use baselines to reduce the variance of their estimates. In the bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between., 'variance' is a fitted model's sensitivity to the particular training sample.

Across five seeds the score varied with a standard deviation of 1.3 points, so a 0.5-point gain means little.