Mathematics & StatisticsGenerative MediaMachine Learning

variational inference

nounAlso: VI, variational Bayes, variational approximation, mean-field approximation, stochastic variational inference, amortized inference

In plain wordsA way of approximating a hard-to-compute probability distribution by picking the closest match from a simpler family of distributions, turning a hard problem of adding up every possibility into an optimization problem.

A family of methods that stand in for an intractable posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces. (or other distribution) p with the best-fitting distribution q of a simpler form — fully factorized ('mean-field'), Gaussian, or parameterized by a neural network — where best-fitting means having the smallest Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right. KL(q ‖ p). Because that divergence involves the unknown normalizing constant, one instead maximizes the evidence lower boundevidence lower bound, n.: A quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn. (ELBO), which falls short of the log marginal likelihood by exactly the KL gap. VI turns inference into optimizationoptimization, n.: Finding the best possible choice according to some score, such as the lowest error or the highest reward. Training an AI is optimization: adjusting its settings until its mistakes are as small as possible., so it scales with stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset. and mini-batches (stochastic VI) and can be amortized by an encoder network that maps each observation to its approximate posterior, as in the variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples.. It is usually much faster than Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability. but biased: minimizing KL(q ‖ p) is mode-seeking and tends to underestimate uncertainty. VI also underlies Bayesian neural networks, topic models such as latent Dirichlet allocationlatent Dirichlet allocation, n.: A popular topic-finding method that assumes every document is a blend of a few themes and every theme favors certain words, then works backward from the words to find both. and the variational view of diffusion modelsdiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static..
Variational inference fit in ten minutes; the MCMC run would have taken two days.

\log p(x) = \underbrace{\mathbb{E}_{q(z)}\big[\log p(x, z) - \log q(z)\big]}_{\text{ELBO}} + \mathrm{KL}\big(q(z)\,\Vert\,p(z \mid x)\big)

Origin

The name comes from the calculus of variations, which optimizes over functions. Mean-field approximations from statistical physics were applied to neural networks by Carsten Peterson and James Anderson (1987); Michael Jordan, Zoubin Ghahramani, Tommi Jaakkola and Lawrence Saul's 1999 tutorial systematized VI for graphical models, and Diederik Kingma and Max Welling's variational autoencoder (2013) made amortized, gradient-based VI standard in deep learning.

Sources

Jordan, M. I., Ghahramani, Z., Jaakkola, T. S. & Saul, L. K. (1999). An Introduction to Variational Methods for Graphical Models. Machine Learning, 37.
Blei, D. M., Kucukelbir, A. & McAuliffe, J. D. (2017). Variational Inference: A Review for Statisticians. Journal of the American Statistical Association, 112(518).

Mentioned in

Bayesian inferenceBayesian inference, n.: Reasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability., Bayesian networkBayesian network, n.: A diagram of causes and effects with probabilities attached — say, how rain and a sprinkler can each make grass wet — that lets a computer update its beliefs about hidden causes when it sees evidence., expected valueexpected value, n.: The long-run average you would get if a chancy situation could be repeated many times — 3.5 for a fair die, even though no single roll shows 3.5., inference engineinference engine, n.: The part of an AI system that does the working-out: in older AI, the program that applies rules to facts; in today's AI, the software that runs a trained model to produce answers., Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right., Monte Carlo methodMonte Carlo method, n.: Estimating an answer by running many random trials and averaging the results — like estimating the odds in a card game by dealing thousands of hands instead of calculating., posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces., probabilistic graphical modelprobabilistic graphical model, n.: A way of describing how many uncertain quantities depend on one another by drawing them as a network, which makes it practical for a computer to reason about all of them together., Bayesian neural networkBayesian neural network, n.: A neural network that keeps a whole range of plausible settings for its internal numbers instead of one best guess, so that it can say how unsure it is about each prediction., belief propagationbelief propagation, n.: A way for a network of uncertain quantities to reach conclusions by having each node pass short messages to its neighbors about what it currently believes, until the beliefs settle down., Jensen's inequalityJensen's inequality, n.: A rule saying that for a bowl-shaped curve, the average of the outputs is at least the output of the average. It is the hidden step behind many proofs in machine learning., Markov chain Monte CarloMarkov chain Monte Carlo, n.: A family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability., Michael I. JordanMichael I. Jordan, prop. n.: An American computer scientist and statistician who did much to bring probability and statistics into machine learning, and who trained many of the field's leaders., probabilistic programmingprobabilistic programming, n.: Writing a statistical model as an ordinary-looking program that includes random choices, and letting the system work out automatically which hidden values best explain the observed data., uncertainty quantificationuncertainty quantification, n.: Measuring how unsure an AI system's predictions are, and why, so that people know when to trust an answer and when to double-check it.

variational inference

Origin

Related

Sources

Mentioned in