generative model
In plain wordsA model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them.
A model of the probability distribution of the data itself — p(x), or p(x | c) given a condition c such as a text prompt — rather than only of a label given an input. A trained generative model can be sampled to produce new data, and some families can also score how probable an input is. Deep generative models differ in how they represent and fit that distribution: autoregressive modelsautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back. factor it into a chain of next-element predictions; variational autoencodersvariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples. and normalizing flowsnormalizing flow, n.: A generator built from a chain of reversible steps that reshape simple random noise into realistic data; because every step can be undone, it can also calculate exactly how likely any example is. map data to and from latent variables; generative adversarial networksgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people. learn a sampler through a contest with a discriminator; and diffusiondiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static. and flow matchingflow matching, n.: A way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples. models learn to turn noise into data step by step. Large language modelslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code. are generative models of text, and most image, video and audio generators combine several of these ideas.
We trained a generative model of normal sensor traces so we could simulate rare failures.
Origin
The generative–discriminative contrast comes from statistical pattern recognition. Neural 'generative models' were prominent by the mid-1990s (the Helmholtz machine, 1995), and Andrew Ng and Michael Jordan's widely cited 2001 comparison of naive Bayes with logistic regression fixed the pairing in textbooks. Deep generative modeling took off with the variational autoencoder (2013) and the generative adversarial network (2014), and reached the public through 'generative AI' after 2022.
Usage note
Not the same as generative AIgenerative AI, n.: AI that produces something new on request, such as an essay, a picture, a song or a piece of code, having absorbed from enormous numbers of existing examples what such things typically look like., which names products and applications; a generative model can be small and classical, such as a naive Bayesnaive Bayes, n.: A simple, fast way of sorting items into categories using probabilities, which pretends that each clue is independent of the others. It is the classic technique behind early email spam filters. classifier or a Gaussian mixture modelGaussian mixture model, n.: A model that explains data as coming from several overlapping bell-shaped clouds, working out where each cloud sits, how spread out it is, and how likely each point is to belong to it..
Related
- generative AIgenerative AI, n.: AI that produces something new on request, such as an essay, a picture, a song or a piece of code, having absorbed from enormous numbers of existing examples what such things typically look like.
- discriminative modeldiscriminative model, n.: A model that learns only to tell things apart — spam or not spam, cat or dog — without learning how to produce new examples of those things.
- diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
- generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people.
- variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples.
- autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back.
- density estimationdensity estimation, n.: Working out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare.
- latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates.
- discriminative modeldiscriminative model, n.: A model that learns only to tell things apart — spam or not spam, cat or dog — without learning how to produce new examples of those things.
Sources
- Ng, A. Y. & Jordan, M. I. (2002). On Discriminative vs. Generative Classifiers: A comparison of logistic regression and naive Bayes. Advances in Neural Information Processing Systems 14 (NIPS 2001).
- Dayan, P., Hinton, G. E., Neal, R. M. & Zemel, R. S. (1995). The Helmholtz Machine. Neural Computation 7(5).
Mentioned in
diffusion modeldiffusion model, n.: An AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static., generative AIgenerative AI, n.: AI that produces something new on request, such as an essay, a picture, a song or a piece of code, having absorbed from enormous numbers of existing examples what such things typically look like., modelmodel, n.: A simplified working version of something. In AI it usually means the trained program, a big set of learned numbers, that takes an input and produces an answer, prediction or piece of content., text-to-image modeltext-to-image model, n.: An AI system that creates a picture from a written description, such as 'a fox reading a newspaper, in watercolor', usually within seconds., text-to-video modeltext-to-video model, n.: An AI that creates a short video clip — increasingly with matching sound — from a written description, such as 'a paper boat sailing down a rainy street'., world modelworld model, n.: An AI's internal simulation of how its world works, which lets it predict what will happen next — especially in response to its own actions — and so plan, imagine or practice without acting for real., autoregressive modelautoregressive model, n.: A model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back., conditional random fieldconditional random field, n.: A statistical method for labeling every item in a sequence, such as marking each word in a sentence as a name, a place or an ordinary word, by choosing all the labels together rather than one at a time., density estimationdensity estimation, n.: Working out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare., discriminative modeldiscriminative model, n.: A model that learns only to tell things apart — spam or not spam, cat or dog — without learning how to produce new examples of those things., generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people., linear discriminant analysislinear discriminant analysis, n.: A statistical method for telling known groups apart. It finds the weighted mix of measurements on which groups differ most relative to their spread, then uses it to sort new items or compress many measurements into a few., naive Bayesnaive Bayes, n.: A simple, fast way of sorting items into categories using probabilities, which pretends that each clue is independent of the others. It is the classic technique behind early email spam filters., posteriorposterior, n.: What you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces., predictionprediction, n.: A model's output for a given input: its best guess at an unknown answer. In AI it need not be about the future; labeling a photo 'cat' counts as a prediction., probability distributionprobability distribution, n.: A complete description of how likely each possible outcome is — for a fair die, one-sixth for each face. A language model produces one of these over its whole vocabulary at every step., samplingsampling, n.: Picking some items at random from a larger set or from a range of possibilities — surveying 1,000 voters, say, or letting an AI choose its next word by a weighted lottery., unsupervised learningunsupervised learning, n.: Machine learning without answer keys: the computer looks at unlabeled data and finds structure on its own, such as groups of similar customers or the main ways in which data vary., variational autoencodervariational autoencoder, n.: A neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples., WaveNetWaveNet, prop. n.: A DeepMind model that generates speech and other sounds one tiny audio sample at a time, producing voices far more natural than earlier computer speech., consistency modelconsistency model, n.: A generator trained to jump from noise straight to a finished image in one step, instead of the dozens of small steps a diffusion model normally takes., Diederik KingmaDiederik Kingma, prop. n.: A Dutch computer scientist who co-invented the variational autoencoder and the Adam optimizer, two of the most widely used tools for training neural networks., generative agentgenerative agent, n.: A character in a computer simulation, run by a language model, that keeps a diary-like memory, draws lessons from it and schedules its days, so that a group of them can act like a plausible small community., Harold CohenHarold Cohen, prop. n.: A British-born painter who wrote AARON, a program that made its own drawings and paintings from the early 1970s, making him a pioneer of art produced by AI.