confidence interval
In plain wordsA range of plausible values for something being estimated, such as 'accuracy 82 percent, give or take 2 points', reflecting the uncertainty that comes from measuring only a sample.
An interval computed from sample data by a procedure that, over repeated sampling, would contain the true value of a parameter a specified fraction of the time — the confidence level, often 95 percent. A typical approximate interval is the estimate plus or minus about two standard errors, justified by the central limit theoremcentral limit theorem, n.: The mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like.; the bootstrap supplies intervals when no formula is convenient. The confidence attaches to the procedure, not to any single interval: a computed 95 percent interval either contains the true value or does not. This differs from the Bayesian credible interval, a range holding 95 percent of posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces. probability. In AI evaluation, confidence intervals on benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time. scores, win rates and pass@kpass@k, n.: A score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code. estimates show whether differences between models exceed sampling noise; leaderboards such as LMArenaLMArena, prop. n.: A public website that ranks AI chatbots by blind taste test: visitors put one question to two unnamed models, pick the reply they prefer, and millions of such votes produce the rankings. have reported them, and models whose intervals overlap substantially may not differ meaningfully.
On 1,000 questions, 82 percent accuracy comes with a 95 percent confidence interval of roughly 79.6 to 84.4.
Origin
Introduced by Jerzy Neyman in a 1934 paper on survey sampling ('On the Two Different Aspects of the Representative Method') and developed in his 1937 theory of estimation, as a frequentist alternative to Ronald Fisher's fiducial intervals. Laplace and Gauss had earlier attached 'probable error' ranges to estimates.
Related
- p-valuep-value, n.: A number saying how surprising your data would be if nothing interesting were going on. A small p-value hints at a real effect, but it is not the probability that the effect is real.
- hypothesis testinghypothesis testing, n.: A formal procedure for deciding whether data show a real effect or could easily be explained by chance — for instance, whether one AI model is genuinely better than another.
- central limit theoremcentral limit theorem, n.: The mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like.
- variancevariance, n.: A number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves.
- benchmarkbenchmark, n.: A fixed, shared test — a set of questions or tasks with an agreed way of scoring — used to compare AI systems with one another and to track progress over time.
- posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces.
Sources
- Neyman, J. (1937). Outline of a Theory of Statistical Estimation Based on the Classical Theory of Probability. Philosophical Transactions of the Royal Society A, 236.
Mentioned in
test settest set, n.: A set of examples kept completely apart from training and used at the end to measure how well a model works on data it has never seen, like a sealed final exam., baselinebaseline, n.: A simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value., bootstrapbootstrap, n.: Estimating how much a result might vary by repeatedly re-drawing samples, with replacement, from the data you already have — like re-running a study many times without collecting new data., central limit theoremcentral limit theorem, n.: The mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like., hypothesis testinghypothesis testing, n.: A formal procedure for deciding whether data show a real effect or could easily be explained by chance — for instance, whether one AI model is genuinely better than another., inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it., p-valuep-value, n.: A number saying how surprising your data would be if nothing interesting were going on. A small p-value hints at a real effect, but it is not the probability that the effect is real., samplingsampling, n.: Picking some items at random from a larger set or from a range of possibilities — surveying 1,000 voters, say, or letting an AI choose its next word by a weighted lottery., conformal predictionconformal prediction, n.: A way to wrap any prediction model so that, instead of one guess, it gives a range or set of answers that contains the truth a chosen share of the time, such as 90%.