law of large numbers

A theorem stating that the average of n i.i.d.i.i.d., abbr.: Short for 'independent and identically distributed': every data point is produced the same way, and none influences any other — like repeated rolls of one fair die. random variablesrandom variable, n.: A quantity whose value depends on chance, such as the number rolled on a die or the next word a chatbot will write. with finite mean converges to their expected valueexpected value, n.: The long-run average you would get if a chancy situation could be repeated many times — 3.5 for a fair die, even though no single roll shows 3.5. as n grows — in probability for the weak law, almost surely for the strong law. It justifies Monte CarloMonte Carlo method, n.: Estimating an answer by running many random trials and averaging the results — like estimating the odds in a card game by dealing thousands of hands instead of calculating. estimation, the use of sample statistics for population quantities, and the founding move of statistical learning theorystatistical learning theory, n.: The mathematics of when and why learning from examples works: how much data is needed, and how well a model trained on some examples will do on new ones.: replacing the true expected lossloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down. with its average over training data (empirical risk minimizationempirical risk minimization, n.: The basic recipe of most machine learning: from the models on offer, pick the one with the smallest errors on the training examples, trusting that score as a guide to how it will do on new data.). Mini-batch gradients approximate full gradients for the same reason, and evaluation on a large benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. estimates true accuracy. The law says nothing about how fast averages converge — the central limit theoremcentral limit theorem, n.: The mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like. and concentration inequalities quantify that — and it can fail for heavy-tailed distributions without a finite mean.

With 10,000 test questions, the law of large numbers lets us treat measured accuracy as a good estimate of the true rate.