prior
In plain wordsWhat you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns.
- 1
Bayesian statistics
The probability distributionprobability distribution, n.: A complete description of how likely each possible outcome is — for a fair die, one-sixth for each face. A language model produces one of these over its whole vocabulary at every step. over hypotheses or parameters representing beliefs before the current data are observed; Bayes' theoremBayes' theorem, n.: A formula for updating how much you believe something after seeing new evidence: weigh your earlier belief by how well that explanation predicts what you actually observed. combines it with the likelihoodlikelihood, n.: A score for a possible explanation of some data: how probable that explanation, or that setting of a model, would have made the data actually seen. to give the posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces.. A prior can carry real knowledge, from earlier studies or domain experts, or be chosen to be weakly informative or nearly flat; conjugate choices, such as a Beta prior on a coin's bias or a Dirichlet prior on categoricalcategorical distribution, n.: The probabilities for a single pick among a fixed set of options, like a weighted die. A language model's guess about the next word is one of these, spread over its whole vocabulary. probabilities, make updating exact and cheap, since the observed counts are simply added to the prior's parameters. In machine learning, a Gaussian prior on weights corresponds to L2 regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data. (weight decayweight decay, n.: A training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data.) and a Laplace prior to L1 under maximum a posteriori estimationmaximum a posteriori estimation, n.: Picking the single most believable setting of a model after combining the evidence with what was assumed beforehand. In practice it often amounts to ordinary training plus a penalty that keeps settings modest., and Gaussian processesGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them. place priors directly on functions. As data accumulate, the likelihood usually dominates and the prior's influence fades.
With only twenty labeled examples, the choice of prior noticeably shifts the posterior.
- 2
General and informal useinformal
By extension, any inductive biasinductive bias, n.: The assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases. or background assumption built into a learner — a convolutional network's 'prior' that nearby pixels are related, a pretrained model's knowledge acting as a prior for fine-tuning — or, informally, a person's existing belief, as in 'update your priors'.
Origin
Jacob Bernoulli distinguished probabilities known a priori from those found a posteriori in Ars Conjectandi (1713), and Thomas Bayes (1763) and Pierre-Simon Laplace reasoned with prior probabilities explicitly. The noun 'prior' became standard in the 20th-century Bayesian revival, for example in Harold Jeffreys's Theory of Probability (1939) and Howard Raiffa and Robert Schlaifer's work on conjugate priors (1961).
Related
- posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces.
- likelihoodlikelihood, n.: A score for a possible explanation of some data: how probable that explanation, or that setting of a model, would have made the data actually seen.
- Bayes' theoremBayes' theorem, n.: A formula for updating how much you believe something after seeing new evidence: weigh your earlier belief by how well that explanation predicts what you actually observed.
- Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability.
- maximum a posteriori estimationmaximum a posteriori estimation, n.: Picking the single most believable setting of a model after combining the evidence with what was assumed beforehand. In practice it often amounts to ordinary training plus a penalty that keeps settings modest.
- inductive biasinductive bias, n.: The assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases.
- regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
- posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces.
Mentioned in
Bayes' theoremBayes' theorem, n.: A formula for updating how much you believe something after seeing new evidence: weigh your earlier belief by how well that explanation predicts what you actually observed., Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability., Gaussian processGaussian process, n.: A way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them., inductive biasinductive bias, n.: The assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases., likelihoodlikelihood, n.: A score for a possible explanation of some data: how probable that explanation, or that setting of a model, would have made the data actually seen., maximum likelihood estimationmaximum likelihood estimation, n.: Tuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this., posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces., regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data., ridge regressionridge regression, n.: Linear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs., Bayesian neural networkBayesian neural network, n.: A neural network that keeps a whole range of plausible settings for its internal numbers instead of one best guess, so that it can say how unsure it is about each prediction., evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn., maximum a posteriori estimationmaximum a posteriori estimation, n.: Picking the single most believable setting of a model after combining the evidence with what was assumed beforehand. In practice it often amounts to ordinary training plus a penalty that keeps settings modest., minimum description lengthminimum description length, n.: The principle that the best explanation of some data is the one that lets you write the data down most briefly, counting both the rule and its exceptions. Good theories compress., Solomonoff inductionSolomonoff induction, n.: A mathematical recipe for ideal prediction: consider every computer program that could have produced the data so far, trust shorter programs more, and predict what they would output next.