Mathematics & Statistics

Vectors, gradients, probability and information: the mathematics under the hood. 108 entries live here, and 353 from other domains touch it.

Essential

correlationA measure of how closely two quantities move together, from −1 (one rises as the other falls) through 0 (no straight-line link) to +1 (they rise together). It does not show that one causes the other.
optimizationFinding the best possible choice according to some score, such as the lowest error or the highest reward. Training an AI is optimization: adjusting its settings until its mistakes are as small as possible.
probabilityHow likely something is, on a scale from 0 for the impossible to 1 for the certain; rolling a six with a fair die scores one-sixth. AI systems state their predictions, even each next word, as probabilities.
vectorAn ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points.

All other entries

104

Bayes' theoremA formula for updating how much you believe something after seeing new evidence: weigh your earlier belief by how well that explanation predicts what you actually observed.
Bayesian inferenceReasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability.
big O notationShorthand for how the cost of a computation grows as the problem gets bigger. Saying attention costs 'O(n²)' means doubling the length of the text roughly quadruples the work.
categorical distributionThe probabilities for a single pick among a fixed set of options, like a weighted die. A language model's guess about the next word is one of these, spread over its whole vocabulary.
causal inferenceWorking out whether one thing actually causes another, rather than merely occurring alongside it — for example, whether a new feature made users stay longer or just happened at the same time.
central limit theoremThe mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like.
chain ruleA rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer.
conditional probabilityThe chance of something once you already know that something else is true — for example, the chance of rain given that the sky is cloudy.
confidence intervalA range of plausible values for something being estimated, such as 'accuracy 82 percent, give or take 2 points', reflecting the uncertainty that comes from measuring only a sample.
confounding variableA hidden third factor that influences two things at once and makes them look linked. Ice-cream sales and drownings both rise in hot weather; neither causes the other.
convex optimizationFinding the lowest point of a bowl-shaped landscape. Because a bowl has only one bottom, such problems can be solved reliably — unlike training a neural network, whose landscape is bumpy.
cosine similarityA score from −1 to 1 for how closely two lists of numbers point the same way, ignoring their size. It is the standard test of whether two texts mean similar things.
counterfactualA statement about what would have happened if something had been different — 'had I left earlier, I would have caught the train'. AI uses counterfactuals to explain decisions and to measure causes.
covarianceA measure of whether two quantities tend to rise and fall together, move in opposite directions, or show no consistent relationship.
cross-entropyA score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
decision theoryThe study of how to make the best choice when outcomes are uncertain, by weighing how likely each result is against how much it matters.
derivativeHow fast something changes as one of its inputs changes — the steepness of a curve at a point. Training an AI means repeatedly following such slopes downhill.
dimensionalityHow many separate numbers are needed to describe something. A point on a map needs two; an AI's description of a single word may use thousands.
do-calculusA set of mathematical rules, devised by Judea Pearl, for working out the effect of deliberately changing something using only observational data plus a diagram of what causes what.
dot productA score of how well two lists of numbers line up, found by multiplying matching entries and adding the products. It is large when the lists point the same way and zero when they are at right angles.
eigenvectorA special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies.
entropyA measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy.
Euclidean distanceHow far apart two points are in a straight line, as a ruler would measure it, extended from two or three dimensions to lists of numbers of any length.
expected valueThe long-run average you would get if a chancy situation could be repeated many times — 3.5 for a fair die, even though no single roll shows 3.5.
Fisher informationA number, or table of numbers, saying how sharply the probability of the data reacts when a model's settings are nudged; the sharper the reaction, the more precisely the data can pin those settings down.
Fourier transformA way of breaking a signal — a sound, an image, any pattern over time or space — into the simple waves of different frequencies that add up to it, much as a prism splits white light into colors.
game theoryThe mathematics of strategic decisions, where each player's best move depends on what the others do. It is used to study competition and cooperation among people, firms, nations and AI agents.
Gödel's incompleteness theoremsTwo famous results from 1931 showing that any consistent set of rules for doing arithmetic leaves some true statements unprovable, and cannot prove that it is itself free of contradictions.
gradientA list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill.
HessianA table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass.
hypothesis testingA formal procedure for deciding whether data show a real effect or could easily be explained by chance — for instance, whether one AI model is genuinely better than another.
i.i.d.Short for 'independent and identically distributed': every data point is produced the same way, and none influences any other — like repeated rolls of one fair die.
importance samplingA shortcut for estimating averages: draw samples from a convenient source, then reweight each one to correct for how much more or less likely it would be under the source you actually care about.
information bottleneckThe idea that a good internal summary of data keeps exactly the details needed for the task and discards everything else, as a good note-taker does.
information theoryA branch of mathematics that treats information as surprise — the less predictable a message, the more it tells you — and works out how far data can be compressed and how reliably it can be sent.
integer programmingFinding the best plan when some decisions must be whole numbers or yes-or-no choices — you cannot open half a warehouse or send 2.7 trucks. That small change makes the best plan far harder to find.
Jaccard similarityA simple score for how much two collections overlap: the number of items they share divided by the number of distinct items in either. Identical sets score 1; sets with nothing in common score 0.
JacobianA table listing how each output of a process responds to a small change in each input. It describes how a transformation locally stretches, squeezes and turns space.
Jensen's inequalityA rule saying that for a bowl-shaped curve, the average of the outputs is at least the output of the average. It is the hidden step behind many proofs in machine learning.
Jensen–Shannon divergenceA balanced, two-way measure of how different two sets of probabilities are, comparing each with their average. Unlike its better-known cousin, it gives the same answer in both directions.
Johnson–Lindenstrauss lemmaA surprising fact about high-dimensional data: a random projection can squash a large set of points into far fewer dimensions while distorting every distance between them only slightly.
Kolmogorov complexityHow briefly something can be described by a computer program that recreates it exactly: a million zeros needs only a tiny program, while a truly random string needs one about as long as itself.
Kullback–Leibler divergenceA score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right.
Lagrange multiplierA mathematical device for finding the best answer when strict rules must be obeyed: each rule becomes an extra price that grows when the rule is broken.
law of large numbersThe mathematical rule that averages of chance events steady out over many repetitions: flip a fair coin a few times and the share of heads can be far from one-half, but over thousands of flips it gets very close.
likelihoodA score for a possible explanation of some data: how probable that explanation, or that setting of a model, would have made the data actually seen.
linear algebraThe branch of mathematics about lists and grids of numbers and the even, straight-line transformations between them. It is the everyday language in which neural networks are written and computed.
linear programmingA mathematical way to find the best use of limited resources, such as a factory's machines or an airline's crews, when every cost and limit scales in simple proportion: twice the output needs twice the input.
linear transformationA rule for moving every point in a space that keeps straight lines straight and the origin in place: rotating, stretching, reflecting or squashing everything evenly. Every grid of numbers describes one.
Lipschitz continuityA guarantee that a function never changes too abruptly: its output can move at most a fixed multiple of how far its input moves.
local minimumA spot lower than everything immediately around it, like the bottom of a small valley, even though a deeper valley may exist somewhere else.
logical inductionA mathematical method for giving sensible, steadily improving probabilities to math claims that haven't been proved or disproved yet, much as a betting market prices uncertain events.
logical uncertaintyUncertainty about math or logic facts that are settled in principle but that you haven't worked out yet — like not knowing a distant digit of pi until you compute it.
log-likelihoodThe logarithm of how probable a model finds the observed data. Taking logs turns a product of many tiny probabilities into a manageable sum; training usually minimizes its negative.
low-rank approximationReplacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation.
manifold hypothesisThe idea that real-world data, such as photos of faces, fills only a thin, smoothly curved sliver of the vast space of all possible data — which is why learning from examples works at all.
Markov chainA chance-driven process that moves between states without memory: the odds of its next move are set by where it is, never by the path that led there. Forecasting tomorrow's weather from today's alone is a simple example.
Markov chain Monte CarloA family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability.
matrixA rectangular grid of numbers arranged in rows and columns. Most of what a neural network has learned is stored in large grids like this.
matrix multiplicationThe rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing.
maximum a posteriori estimationPicking the single most believable setting of a model after combining the evidence with what was assumed beforehand. In practice it often amounts to ordinary training plus a penalty that keeps settings modest.
maximum likelihood estimationTuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this.
minimum description lengthThe principle that the best explanation of some data is the one that lets you write the data down most briefly, counting both the rule and its exceptions. Good theories compress.
Monte Carlo methodEstimating an answer by running many random trials and averaging the results — like estimating the odds in a card game by dealing thousands of hands instead of calculating.
mutual informationHow much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated.
Nash equilibriumA stable standoff in a game: each player's choice is already their best answer to what the others are doing, so nobody gains by changing course alone — though the result need not be good for anyone.
no free lunch theoremA mathematical result saying that no learning or search method is best for every possible problem: any method that wins on some problems must lose on others.
normA way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes.
normal distributionThe familiar bell-shaped curve: most values cluster near the average, and values become rapidly rarer the farther they lie from it. Measurement errors and adult heights roughly follow it.
NP-completeDescribes problems, such as solving giant Sudokus, where a proposed answer is quick to check but no known method finds one quickly; a fast method for any one of them could be turned into one for all.
numerical stabilityWhether a computer calculation stays accurate despite the tiny rounding errors in every step, instead of blowing up into nonsense such as infinities.
objective functionThe score that an optimization process tries to push as low, or as high, as possible. For an AI, it defines what 'doing well' means during training.
Occam's razorA rule of thumb for choosing between explanations: if two fit the evidence equally well, bet on the one that assumes less. In AI it motivates preferring simpler models, which are less likely to fit noise.
orthogonalityThe mathematical version of being at right angles. Two directions are orthogonal when moving along one tells you nothing about the other, like north and east on a map.
PAC learningA mathematical definition of what it means to learn from examples: given enough data, a learner should, with high probability, find a rule that is almost always right.
partial derivativeHow much a quantity changes when you nudge just one of its many inputs while holding all the others fixed.
posteriorWhat you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces.
priorWhat you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns.
probability density functionA curve showing where a measurement is likely to fall. The area under the curve between two values gives the chance of landing between them.
probability distributionA complete description of how likely each possible outcome is — for a fair die, one-sixth for each face. A language model produces one of these over its whole vocabulary at every step.
p-valueA number saying how surprising your data would be if nothing interesting were going on. A small p-value hints at a real effect, but it is not the probability that the effect is real.
random variableA quantity whose value depends on chance, such as the number rolled on a die or the next word a chatbot will write.
rankThe number of truly independent directions a grid of numbers contains. A low-rank grid can be rebuilt exactly from a few simple building blocks, however large it looks.
saddle pointA spot on a surface that is a low point in one direction but a high point in another, like the middle of a horse's saddle or a mountain pass.
samplingPicking some items at random from a larger set or from a range of possibilities — surveying 1,000 voters, say, or letting an AI choose its next word by a weighted lottery.
scalarA single number, such as 3.7 or −2, as opposed to a list or grid of numbers. A model's error score and its learning rate are both scalars.
Shapley valueA rule from game theory for splitting a shared reward fairly among team members, by averaging how much each member adds when joining the team in every possible order.
Simpson's paradoxA statistics trap in which combining groups reverses what each group shows: a treatment can do better than another for men and for women separately, yet worse when everyone is counted together.
singular value decompositionA way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data.
Solomonoff inductionA mathematical recipe for ideal prediction: consider every computer program that could have produced the data so far, trust shorter programs more, and predict what they would output next.
spurious correlationA link in data that fools you: two things move together, yet neither drives the other, because a hidden shared cause, the way the data were collected or luck produced the pattern. AI models readily latch onto such links.
statistical learning theoryThe mathematics of when and why learning from examples works: how much data is needed, and how well a model trained on some examples will do on new ones.
stochasticInvolving chance, so the same starting point can lead to different outcomes. AI chatbots are stochastic: ask the same question twice and you may get two different answers.
structural causal modelA precise description of what causes what in a system — a diagram of arrows plus equations — that predicts what would happen if you intervened, not just what tends to occur together.
Taylor seriesA way of approximating a complicated curve near a chosen point by adding up simple pieces: a constant, a straight-line slope, a bend, and so on.
tensorA block of numbers with any number of dimensions: a single number, a list, a grid, or a stack of grids. AI software stores and processes all its data this way.
Turing machineAn imaginary, extremely simple computer — a read-write head moving along an endless tape — that Alan Turing used in 1936 to define exactly what can be computed at all.
uncertainty quantificationMeasuring how unsure an AI system's predictions are, and why, so that people know when to trust an answer and when to double-check it.
utility functionA way of scoring every possible outcome by how much someone, or some AI, wants it. A rational agent is often modeled as choosing whatever maximizes its expected score.
varianceA number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves.
variational inferenceA way of approximating a hard-to-compute probability distribution by picking the closest match from a simpler family of distributions, turning a hard problem of adding up every possibility into an optimization problem.
VC dimensionA measure of how flexible a family of prediction rules is: the largest number of points it can label in every possible way. More flexible families need more data to learn reliably.
Wasserstein distanceA way of comparing two spreads of probability by imagining one as heaps of sand that must be shoveled into the shape of the other: the less shoveling, weighted by distance, the more alike they are.
Zipf's lawThe pattern that word use is lopsided: in a big text the top word is roughly twice as common as the runner-up and ten times as common as the tenth, so a few words dominate and most are rare.

Also relevant

353

A/B testingComparing two versions of something — two models, two prompts, two designs — by randomly showing each to different groups of real users and measuring which works better.
accumulated local effectsA chart of how a model's predictions change with one input, built from small local comparisons so it stays reliable when inputs are related to each other, unlike simpler average-based charts.
activation spaceThe imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions.
AdaGradA training method that gives each of a model's settings its own step size, shrinking steps for settings that have already had many large updates and keeping them larger for rarely updated ones.
AdamThe most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been.
agent foundationsA research program that tried to work out, with mathematics, how an idealized reasoning agent should think and decide, as groundwork for building AI that stays safe.
AIMEA hard US high-school math contest whose 15 questions each have a whole-number answer; AI labs use each year's new papers to test their models' mathematical reasoning.
AIXIA mathematical description of a perfect learning agent that weighs every possible explanation of its experience and acts to get the most future reward. It cannot actually be built, but it defines an ideal.
Akaike information criterionA score for comparing statistical models that rewards fitting the data well but charges a penalty for every extra adjustable number, to avoid picking needlessly complicated models. Lower is better.
algorithmA precise, step-by-step procedure for solving a problem or doing a task, like a recipe a computer can follow. In the news, 'the algorithm' often means software that ranks what you see online.
AlphaEvolveA Google DeepMind system that combines AI language models with an evolution-like search to invent and improve computer programs, finding new results in mathematics and computing.
Shun-ichi AmariA Japanese mathematician who trained multilayer neural networks by gradient descent as early as 1967 and founded information geometry, the source of the 'natural gradient' method.
ant colony optimizationA method for finding good routes and schedules inspired by how ants find short paths to food: simulated ants leave 'scent' on good paths, which attracts other ants, so the best routes gradually stand out.
approximate nearest neighbor searchA family of shortcuts for finding the most similar items in a huge collection without checking every one — accepting an occasional miss in exchange for answers thousands of times faster.
ARIMAA classic statistical recipe for forecasting a series of numbers over time: remove any steady trend, then predict the next value from recent values and recent prediction errors.
arithmetic intensityHow much calculating a task does for each piece of data it fetches from memory. Tasks with little calculation per fetch are slowed by memory, not by the chip's math speed.
associative memoryA memory that is searched by content rather than by location: given part of a stored pattern, or a noisy copy of it, it recalls the whole, much as a few notes can bring back an entire song.
autoformalizationUsing AI to translate ordinary mathematics written for people — definitions, theorem statements, proofs — into a strict formal language that a proof-checking program can verify.
automated theorem provingGetting computers to prove mathematical or logical statements by themselves, by searching for a chain of valid steps that leads from basic assumptions to the statement.
automatic differentiationA technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically.
autoregressive modelA model that produces a sequence one piece at a time, each new piece depending on the pieces already produced — like writing a sentence word by word without going back.
backpropagationThe method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
baggingTraining many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error.
Bayes error rateThe lowest error any classifier could ever achieve on a task, because some cases are genuinely ambiguous: even a perfect model that knew the true odds would still sometimes guess wrong.
Bayesian information criterionA score for comparing statistical models that, like AIC, rewards good fit and penalizes extra adjustable numbers, but penalizes them more as the dataset grows, favoring simpler models. Lower is better.
Bayesian networkA diagram of causes and effects with probabilities attached — say, how rain and a sprinkler can each make grass wet — that lets a computer update its beliefs about hidden causes when it sees evidence.
Bayesian neural networkA neural network that keeps a whole range of plausible settings for its internal numbers instead of one best guess, so that it can say how unsure it is about each prediction.
Bayesian optimizationA smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next.
belief propagationA way for a network of uncertain quantities to reach conclusions by having each node pass short messages to its neighbors about what it currently believes, until the beliefs settle down.
Bellman equationA rule saying that how good a situation is equals the reward you get now plus how good the next situation is, slightly discounted — so long-term planning can be solved one step at a time.
Richard BellmanAn American mathematician who invented dynamic programming, a way of solving long chains of decisions step by step, which became the mathematical core of reinforcement learning.
benign overfittingWhen a model fits its training data perfectly — even the errors and noise — yet still predicts well on new data, defying the textbook warning that this always backfires.
biasA word with several AI meanings: an adjustable number added inside a model; a systematic error in estimates; or unfair treatment of some groups of people by an AI system.
bias–variance tradeoffThe balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.
BM25A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents.
Boltzmann machineA network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them.
Boolean satisfiability problemThe puzzle of deciding whether a set of true-or-false conditions can all be met at once. It is famously hard in theory, yet modern solvers routinely handle millions of conditions.
boostingBuilding a strong predictor from many weak ones trained one after another, where each new model concentrates on the mistakes the previous ones made.
bootstrapEstimating how much a result might vary by repeatedly re-drawing samples, with replacement, from the data you already have — like re-running a study many times without collecting new data.
bounded rationalityThe idea that real decision-makers, whether people or machines, cannot consider every option or compute the perfect choice, so they make reasonable decisions within limits of time, information and thinking power.
Bradley–Terry modelA simple statistical rule for turning head-to-head comparisons into scores: each item gets a strength, and the chance one beats another depends only on the difference in their strengths.
branch and boundA method for finding the provably best answer to a hard puzzle: divide the possibilities into groups, quickly estimate the best any group could offer, and throw out every group that cannot beat the best answer already in hand.
breadth-first searchA way of exploring a maze or network level by level — first every place one step away, then every place two steps away, and so on — so that the shortest route is found first.
Leo BreimanAn American statistician who invented random forests and bagging, decision-tree methods still in wide use, and urged statisticians to judge models by how well they predict.
Brier scoreA score for probability forecasts: the average squared gap between each forecast probability and what actually happened (1 if it did, 0 if not). Zero is perfect; lower is better.
CARTA classic recipe for building decision trees that splits data two ways at every step, works for both categories and numbers, and then trims the tree back so it does not memorize noise.
causal abstractionA mathematical way to check whether a simple, human-readable recipe really describes what a neural network does inside, by testing whether matching internal parts behave the same way when swapped.
causal mediation analysisA statistical method for splitting a cause's effect into the part that flows through some go-between and the part that does not; in AI it is used to find which internal parts carry a behavior.
centered kernel alignmentA score from 0 to 1 measuring how similar two neural networks' internal representations are, by comparing how each arranges the same set of inputs relative to one another.
certified robustnessA mathematical guarantee — not just a test result — that no change to an input within a set limit can alter a model's answer.
ChinookA checkers program from the University of Alberta that in 1994 became the first program crowned world champion in a contest against people, and in 2007 proved that perfect checkers always ends in a draw.
circuitA small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
coefficient of determinationA score for numerical predictions showing how much of the variation in the true values a model accounts for, compared with always guessing the average: 1 is perfect, 0 is no better than the average.
Cohen's kappaA score for how much two people labeling the same items agree beyond what lucky guessing would produce: 1 means perfect agreement, 0 means no better than chance.
combinatorial explosionThe way the number of possible combinations — moves, routes, schedules — grows so quickly as a problem gets bigger that checking them all becomes impossible, even for the fastest computers.
computational graphA map of a calculation drawn as boxes and arrows: each box is a simple operation and each arrow carries a number to the next step. AI software uses it to trace errors backward.
conditional random fieldA statistical method for labeling every item in a sequence, such as marking each word in a sentence as a name, a place or an ordinary word, by choosing all the labels together rather than one at a time.
configuration spaceAn imaginary space in which a robot's whole body position is a single point, which turns 'move this many-jointed robot without hitting things' into 'find a path for a dot'.
conformal predictionA way to wrap any prediction model so that, instead of one guess, it gives a range or set of answers that contains the truth a chosen share of the time, such as 90%.
constraint satisfaction problemA problem in which each item must be given a value so that every rule is obeyed, like filling in a Sudoku, coloring a map so no neighbors match, or building a timetable without clashes.
construct validityWhether high marks on a test actually reflect the skill the test is named after — for example, whether a 'reasoning' test rewards reasoning rather than memory or test-taking tricks.
control theoryThe mathematics of making machines behave as intended by constantly measuring what they are doing and correcting them, as a thermostat, cruise control or self-balancing robot does.
convergenceThe point at which training settles down and further steps no longer meaningfully improve the model — and, in theory, a guarantee that a method will reach a solution, and how fast.
convolutionAn operation that slides a small grid of weights across an image or signal, multiplying and adding at each spot, to measure how strongly a pattern appears at every location.
cooperative inverse reinforcement learningA model of a human and an AI working as a team, where only the human knows the goal and the AI must work it out while helping.
counterfactual fairnessA fairness test that asks: had this person been of another race or sex, with everything that would have followed from that, would the system have decided the same way?
counterfactual regret minimizationA method for working out near-unbeatable strategies in games with hidden information, like poker, by repeatedly playing against itself and shifting each decision toward the moves it most regrets not making.
covariate shiftA kind of mismatch where the inputs an AI system sees change — say, photos from a new camera — while the right answer for any given input stays the same.
cross-validationA fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn.
curse of dimensionalityThe problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much.
data scienceThe practice of collecting, cleaning, analyzing and interpreting data, using statistics, programming and machine learning, to answer questions and guide decisions.
decision boundaryThe dividing line, or surface, where a classifier switches from predicting one category to another. Simple models draw straight lines; flexible ones draw curves.
degrees of freedomThe number of independent ways something can move. A robot arm with six motorized joints has six; a free-floating object can slide three ways and turn three ways.
Dempster–Shafer theoryA theory of reasoning with uncertain evidence that lets you hold back belief — in effect saying 'I don't know' — instead of being forced to divide all your confidence among the possibilities.
density estimationWorking out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare.
depth-first searchA way of exploring a maze or network by following one path as far as it goes, then backing up to the most recent fork and trying the next branch.
description logicA family of precise logical languages for defining categories and how they relate, such as 'a parent is a person with at least one child', designed so that a computer's reasoning about them always finishes.
developmental interpretabilityThe study of how a neural network's internal structure forms during training — when and how new abilities and mechanisms appear — rather than only examining the finished model.
dictionary learningA method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts.
differential privacyA mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person.
diffusion modelAn AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
diffusion samplerThe step-by-step routine a diffusion model follows to turn noise into a finished picture; the choice of sampler and number of steps trades speed against quality.
Dijkstra's algorithmA classic method for finding the shortest routes from one starting point to every other point on a map or network, by always settling the nearest not-yet-settled point next.
dimensionality reductionSqueezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from.
directional ablationErasing one specific direction from an AI model's internal signals while it runs, so whatever that direction encodes — such as the tendency to refuse — can no longer influence what the model does.
discount factorA number between 0 and 1 that makes an AI agent value future rewards less than immediate ones — like preferring a dollar today to a dollar next year.
double descentA surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse.
DP-SGDA way of training a neural network so that it cannot learn too much about any single person in the data: each example's influence is capped, and random noise is added at every step.
Cynthia DworkAn American computer scientist who co-invented differential privacy, the leading mathematical definition of privacy in data analysis, and helped found the study of fairness in algorithms.
dynamic programmingA problem-solving method that breaks a big problem into smaller overlapping pieces, solves each piece once and reuses the answers — the basis for computing the best long-term strategy step by step.
edge of stabilityA surprising pattern in how neural networks train: the error landscape keeps getting steeper until it is just steep enough to make training wobble, then stays balanced at that edge while the error still falls.
elastic netA blend of two ways of keeping a model simple: it can drop useless inputs entirely while treating groups of similar inputs alike, instead of picking one of them at random.
Elo ratingA number that ranks players — or AI models — from the results of head-to-head contests, rising when you beat a strong opponent and falling when you lose to a weak one.
embedded agencyThe puzzle of how an agent should reason when it is itself part of the world it is trying to understand and change, rather than standing outside it.
embeddingA list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
empirical risk minimizationThe basic recipe of most machine learning: from the models on offer, pick the one with the smallest errors on the training examples, trusting that score as a guide to how it will do on new data.
energy-based modelA model that scores how well things fit together by giving each possibility an 'energy': low for plausible combinations, high for implausible ones, like a ball settling into the lowest valleys.
entropy regularizationRewarding an AI agent for keeping some randomness in its choices, so it keeps exploring alternatives instead of locking in on one behavior too early.
equivarianceA property of a model whose output changes in step with its input: move the cat in a photo, and the network's internal 'cat' signal moves the same way.
evidence lower boundA quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn.
evolutionary computationProblem-solving by simulated evolution, much as breeders improve crops: keep many candidate answers, let the better ones produce slightly altered offspring, drop the worse, and repeat for many rounds.
expectation–maximization algorithmA method for fitting a model when some information is hidden: guess the hidden parts using the current model, refit the model as if those guesses were right, and repeat until nothing changes.
exponential smoothingA simple forecasting method that predicts the next value as a weighted average of past values, giving the most weight to recent ones and steadily less to older ones.
fairness impossibility theoremA mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once.
false negativeA miss: a test or AI system gives the all-clear, for example no illness, no fraud or no danger, when the problem is in fact there.
false positiveA false alarm: a test or AI system raises a flag, for example for illness, fraud or danger, when nothing of the kind is really there.
first-order logicA precise formal language for statements about objects and their relations, using 'for all' and 'there exists', such as 'every person has a mother'. It is the standard logic of mathematics and of classic AI.
floating-point formatA way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster.
FLOPA single arithmetic step, such as multiplying or adding two numbers. Counting them measures how much computing went into an AI model, and laws use such counts to decide which models face extra rules.
flow matchingA way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples.
forward kinematicsThe math that works out where a robot's hand or tool ends up, given the angles of all its joints, by chaining together the movement each joint adds.
Fréchet inception distanceA score for image generators that compares the overall statistics of their pictures with those of real photos, as seen by an image-recognition network; lower is better.
free energy principleA sweeping theory by the neuroscientist Karl Friston that living things persist by keeping themselves in expected states — in effect, by constantly minimizing surprise about what they sense.
Jerome FriedmanAn American statistician who invented gradient boosting, the method behind XGBoost and other leading tools for spreadsheet-style data, and co-wrote a standard machine-learning textbook.
Karl FristonA British neuroscientist who proposed that brains, and perhaps all living systems, work by constantly predicting their surroundings and acting to reduce surprise, an idea some see as a blueprint for AI.
FrontierMathSeveral hundred new math problems, kept unpublished so AI systems cannot have trained on them, written by professional mathematicians to check whether machines can do advanced, even research-level, mathematics.
FunSearchA Google DeepMind method that discovered new mathematical constructions by having a language model repeatedly rewrite short computer programs, keeping the versions that scored best.
fuzzy logicA kind of logic that allows degrees of truth, so a statement like 'the room is warm' can be 70 percent true, which helps machines handle vague everyday categories.
g factorThe common factor behind the finding that people who do well on one kind of mental test tend to do well on others. Researchers also look for a similar general factor across AI models.
game treeA branching diagram of every way a game could unfold: each point is a position, each branch a possible move, and the tips of the branches are wins, losses or draws.
Gaussian mixture modelA model that explains data as coming from several overlapping bell-shaped clouds, working out where each cloud sits, how spread out it is, and how likely each point is to belong to it.
Gaussian processA way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them.
generalizationHow well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers.
generalized additive modelA prediction model that adds up separate, possibly curved effects of each input, so each input's influence can be drawn as its own graph and read off directly.
generalized linear modelA family of statistical models that extends straight-line regression to outcomes such as yes/no answers, counts or waiting times, by linking the average outcome to a weighted sum of inputs through a suitable curve.
generative modelA model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them.
genetic algorithmA problem-solving method that copies natural evolution: many candidate solutions compete, the best are 'bred' by mixing and randomly tweaking them, and over many generations the solutions improve.
geometric deep learningAn approach to designing neural networks around the shapes and symmetries of the data, whether grids, graphs, surfaces or 3-D structures, so the network respects how that data can be moved or rearranged.
Gini impurityA score for how mixed a group is: zero when every item belongs to the same category, higher when categories are jumbled together. Decision trees choose splits that make groups purer.
I. J. GoodA British statistician who broke codes with Alan Turing and in 1965 argued that a machine smarter than us could design still smarter machines, setting off an 'intelligence explosion'.
gradient boostingA method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it.
gradient descentA method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.
GSM8KA set of about 8,500 grade-school math word problems, each needing a few steps of simple arithmetic, used to test whether language models can reason through a problem step by step.
Gumbel-softmaxA mathematical trick that lets a neural network make a choice among separate options, such as picking one word or one category, while still learning through small, smooth adjustments.
hill climbingA simple search method that keeps making whichever small change improves things, like climbing in fog by always stepping uphill — which can leave it stuck on a small hill instead of the highest peak.
hinge lossA way of scoring a classifier's mistakes that charges nothing for answers that are right with room to spare, and charges more and more the further an answer falls on the wrong side.
HNSWA popular way to organize millions of embeddings for fast similarity search: a layered web of links where the search starts with long jumps on sparse top layers, then takes ever shorter steps.
homomorphic encryptionA way of locking data so that a computer can still work with it while it stays locked: the computer never sees the real numbers, and only the owner can open the answer.
Hopfield networkA network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint.
Horn clauseA simple, restricted form of logical rule ('if these conditions all hold, then this one conclusion holds') that computers can reason with very efficiently.
Huber lossA way of measuring prediction errors that treats small errors like squared distance but large ones more gently, so a few wild data points cannot drag the whole model off course.
Marcus HutterA German computer scientist who defined AIXI, a mathematical ideal of a perfectly intelligent agent, and who argues that compressing data well is a measure of intelligence.
hypothesis spaceThe full menu of possible models a learning method is allowed to choose from — for example, all straight lines. Learning means picking the item on the menu that best fits the data.
IMO Grand ChallengeThe goal of building an AI that earns a gold medal at the International Mathematical Olympiad, the top contest for school-age mathematicians. AI systems first reached gold-medal scores in 2025, though not under the challenge's strict original rules.
implicit regularizationThe hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions.
imputationFilling in missing values in a dataset with sensible estimates, such as the typical value or a prediction from the other columns, so the data can be analyzed or used for training.
independent component analysisA technique for unmixing signals that have been blended together, such as separating individual voices from several microphones recording a crowded room.
individual fairnessThe idea that a fair system should treat similar people similarly: two applicants who are alike in every way that matters should get similar outcomes.
inductive biasThe assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases.
inferenceDrawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it.
influence functionA mathematical shortcut for estimating how much a single training example shaped a model's behavior, for instance how a prediction would change if that example had been left out, without retraining the model.
information gainHow much a question reduces uncertainty about the answer. A decision tree asks first the question that removes the most uncertainty, measured in bits.
integrated gradientsA way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way.
integrated information theoryA scientific theory that measures consciousness by how much a system works as a single, unified whole rather than as separate parts. It implies that ordinary computers, whatever software they run, have little or no consciousness.
interactive theorem provingWriting a proof in a precise computer language so that software checks every step: the person supplies the ideas, the computer fills in small steps and guarantees that nothing is wrong.
inter-annotator agreementA measure of how often different people labeling the same data give the same answer. Low agreement warns that the task is unclear or the labels are unreliable.
International Mathematical OlympiadThe world's top math competition for high-school students. Because its problems demand creative written proofs, not just answers, earning a gold medal there became a famous test of AI reasoning, first passed in 2025.
intersection over unionA score from 0 to 1 for how well two shapes overlap, such as a predicted box and the true box around an object: the shared area divided by the combined area.
invarianceA property of a model whose answer does not change when the input is altered in an irrelevant way, such as a photo classifier that still says 'cat' when the cat moves or the picture is flipped.
inverse kinematicsWorking backward from where you want a robot's hand to be to the joint angles that will put it there, like working out how to bend your arm to touch your nose.
irreducible lossThe part of a model's error that no amount of extra size, data or training can remove, because the thing being predicted is itself partly random — like the unpredictability of the next word in an ordinary sentence.
IsabelleA widely used proof-checking program in which people write mathematics and verified software; it is known for automation that calls other theorem provers to finish routine steps.
item response theoryA set of statistical models from educational testing that work out both how able each test-taker is and how hard each question is from who answered what correctly; now also used to evaluate AI models.
Michael I. JordanAn American computer scientist and statistician who did much to bring probability and statistics into machine learning, and who trained many of the field's leaders.
Kalman filterA method that keeps a running best guess of something you cannot measure exactly, such as a vehicle's position, by blending predictions from a motion model with noisy sensor readings.
kernelA word with several AI meanings: a similarity measure used by some learning methods; a small grid of numbers that scans an image inside a neural network; or a small program that runs on a GPU.
kernel trickA mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space.
KL penaltyA cost added during training that grows as an AI model drifts away from its starting version, so it improves at the goal without forgetting how to write normally or gaming its scorer.
k-means clusteringA simple way to split data into a chosen number of groups: guess group centers, assign every point to its nearest center, move each center to the middle of its points, and repeat.
Daphne KollerAn Israeli-American computer scientist who wrote the standard text on probabilistic reasoning in AI, co-founded the online-course company Coursera, and founded a machine-learning drug-discovery company.
label shiftA kind of mismatch where the mix of answers changes — say, a disease becomes far more common during an outbreak — while each individual case still looks the same as before.
language modelA system that has learned how likely different sequences of words are, so it can predict the next word in a text and, by doing that repeatedly, write text of its own.
lassoA version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model.
latent Dirichlet allocationA popular topic-finding method that assumes every document is a blend of a few themes and every theme favors certain words, then works backward from the words to find both.
latent spaceThe hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates.
LEACEA one-step mathematical method for scrubbing a concept, such as gender, from an AI model's internal data so that no simple detector can find it, while changing everything else as little as possible.
LeanA programming language and proof checker in which mathematicians write proofs that the computer verifies line by line. It has become the main language in which AI systems produce checkable proofs.
learningGetting better at something through experience. For a machine, it means changing its internal settings based on data so that it performs a task better than before.
LibratusA poker-playing AI from Carnegie Mellon University that in 2017 decisively beat four top professionals at two-player no-limit Texas hold'em. Its successor, Pluribus, beat professionals at six-player poker in 2019.
linear discriminant analysisA statistical method for telling known groups apart. It finds the weighted mix of measurements on which groups differ most relative to their spread, then uses it to sort new items or compress many measurements into a few.
linear regressionFitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest.
linear representation hypothesisThe idea that a neural network stores many concepts as simple directions inside its internal activity, so how much of a concept is present can be read off by measuring along its direction.
linear separabilityWhen two groups of points can be perfectly divided by a single straight line or, in more dimensions, a flat surface. Many simple learning methods only fully succeed in this case.
Seppo LinnainmaaA Finnish computer scientist who in 1970 first published the method for computing all of a program's derivatives in one backward pass, the mathematical core of backpropagation.
local differential privacyA privacy method in which each person's device scrambles their data with random noise before sending it anywhere, so the company collecting it never sees anyone's true answer but can still estimate overall trends.
local learning coefficientA number measuring how complex a trained neural network effectively is around its current settings — usually far less than its raw parameter count suggests — used to spot changes as the network learns.
local searchA family of problem-solving methods that start with some complete answer, however poor, and keep improving it by small changes, instead of building an answer piece by piece from scratch.
local SGDA way to train on many machines that talk to each other only occasionally: each trains its own copy for a while, then they average their copies and continue, cutting the amount of communication needed.
locality-sensitive hashingA trick for finding look-alikes in a huge collection without comparing every pair: items get short codes designed so that similar items tend to share a code, and only items sharing one are compared.
log lossA score for probability predictions that punishes confident mistakes severely: saying '99% sure' and being wrong costs far more than saying '60% sure' and being wrong. Lower is better.
log probabilityThe logarithm of the probability a model gives to a token or a text. Computers use it because multiplying many tiny probabilities runs into rounding problems, whereas adding their logarithms does not.
logistic regressionA simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve.
logitA raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity.
long tailThe huge number of rare things — uncommon words, unusual road situations, obscure facts — that each appear seldom but together make up a large share of what an AI system meets.
loss functionA score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down.
loss landscapeA way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys.
Lucas–Penrose argumentAn argument that human minds cannot be computers because, using a famous result in logic, people can see that certain statements are true that no fixed computer program could prove.
marginHow far, and on which side, an example lies from a classifier's dividing line. A wide gap between the line and the nearest examples suggests the classifier will handle new data well.
Markov decision processA mathematical description of a step-by-step decision problem: each step you pick an action, receive a reward and then land in a new situation, drawn at random from odds that depend only on the current situation and your choice.
Markov propertyThe assumption that the future depends only on the present situation, not on how you got there — so knowing the current state is enough to predict what happens next.
Markov random fieldA model of many linked uncertain quantities, such as neighboring pixels in a photo, in which each one depends directly only on its neighbors, with no notion of which one causes which.
MATHA set of 12,500 problems from high-school math competitions, each with a full worked solution, used to test language models on genuinely challenging mathematics.
MathlibA huge shared library of mathematics written in the Lean language, in which every definition and proof has been checked by computer. Mathematicians keep extending it, and AI proof systems are trained on it.
matrix factorizationBreaking a big table of numbers, such as which users liked which films, into two smaller tables of hidden traits, so missing entries can be filled in by multiplying them back together.
maximal update parametrizationA recipe for scaling a neural network's starting values and learning speeds with its size so that settings tuned on a small, cheap model work unchanged on a huge one.
mean absolute errorA way of scoring numerical predictions: average how far off each prediction is, ignoring direction, so the score reads as 'typically off by this much'. Lower is better.
mean squared errorA way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better.
mel spectrogramA version of a sound picture whose pitch axis is squeezed to match human hearing, giving more detail to low sounds than to high ones; the usual way AI models take in audio.
metaheuristicA general-purpose strategy for finding good answers to hard problems, such as imitating evolution or cooling metal, that can be adapted to many different problems instead of being designed for just one.
MFCCA short list of numbers, usually about 13, that summarizes the shape of a brief slice of sound in a way that mimics human hearing; for decades the standard input to speech recognizers.
mild optimizationBuilding an AI that settles for a solid result instead of squeezing out every last bit of score, since the most extreme ways to max out a goal are often loopholes or excesses nobody wanted.
MinHashA shortcut for spotting near-copies among millions of documents: each document is boiled down to a short fingerprint, and similar documents end up with similar fingerprints.
miniF2FA test of 488 math problems, mostly from competitions, written in a form a computer can check, used to measure how well AI systems produce proofs that are verified automatically.
minimaxA way for a game-playing program to choose moves by assuming the opponent will always make their best reply, and picking the move whose worst-case outcome is best.
mode connectivityThe discovery that two separately trained versions of a network, though their settings look very different, can usually be joined by a path along which the model keeps working well.
modelA simplified working version of something. In AI it usually means the trained program, a big set of learned numbers, that takes an input and produces an answer, prediction or piece of content.
model capacityHow much a model is able to learn: the variety and complexity of patterns it can represent or memorize. Too little and it misses real patterns; very large models can even memorize pure noise.
model predictive controlA control method that repeatedly plans the best moves a few seconds ahead using a model of the machine, carries out only the first move, then plans again with fresh measurements.
model selectionChoosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best.
momentumA training trick that lets each update keep part of the previous update's direction, like a ball rolling downhill that builds up speed and coasts over small bumps.
motion planningWorking out how a robot should move from where it is to where it needs to be, without hitting anything and within the limits of what its body can do.
multi-agent reinforcement learningReinforcement learning with several AI agents learning at once in a shared world, where each must adapt to the others — as teammates, rivals or both.
multi-armed banditA puzzle named after a gambler choosing between slot machines: you must keep picking among options whose payoffs are unknown, learning which is best while wasting as few tries as possible on the others.
naive BayesA simple, fast way of sorting items into categories using probabilities, which pretends that each clue is independent of the others. It is the classic technique behind early email spam filters.
neural collapseA striking tidiness that appears when classifiers are trained for a long time: the network's internal summaries of each class shrink to single points arranged as evenly and as far apart as possible.
neural operatorA neural network that learns how a whole physical system responds, such as how a fluid flows given its starting state, so it can stand in for slow simulations and answer new cases quickly.
neural ordinary differential equationA neural network that, instead of passing data through a fixed stack of layers, describes how the data changes continuously over 'time' and uses an equation solver to follow that change.
neural tangent kernelA mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior.
n-gram modelAn early type of language model that predicts the next word from just the few words before it, using counts of how often each word followed those words in a large collection of text.
nonparametric modelA model that lets the data decide how complex it becomes: the more examples it gets, the more detail it can capture — for instance, one that predicts by comparing a new case with every stored example.
normalized discounted cumulative gainA score for a ranked list of results that rewards putting the most relevant items near the top, scaled so that a perfect ordering scores 1.
normalizing flowA generator built from a chain of reversible steps that reshape simple random noise into realistic data; because every step can be undone, it can also calculate exactly how likely any example is.
NumPyThe basic free Python library for working quickly with large grids of numbers; nearly every data-science and machine-learning tool in Python is built on it or copies its style.
off-switch gameA simple game-theory model showing that a robot unsure what its human wants has a reason to let the human switch it off — and loses that reason once it is sure.
online learningLearning from data that arrive one piece at a time, updating the model after each new example instead of training once on a fixed collection, as a spam filter does while emails stream in.
optimal controlFinding the best way to steer a system over time, such as the most fuel-efficient path for a rocket or the smoothest motion for a robot arm, by minimizing a cost while obeying the laws of motion.
optimizerThe rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment.
ordinary least squaresThe standard way to fit a line through data: choose the line that makes the total of the squared vertical distances from the points to the line as small as possible.
overparameterizationGiving a model far more adjustable settings than it would seem to need — often more than it has training examples. Classic theory predicted this would backfire; in deep learning it usually helps.
parameterOne of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned.
parametric modelA model whose shape is fixed in advance by a set number of adjustable settings, like a straight line's slope and intercept. More data refines those settings but never adds new ones.
partial dependence plotA chart showing how a model's average prediction changes as one input is varied while everything else stays as it is in the data — for example, predicted house price against floor area.
partially observable Markov decision processA version of the standard decision-making model in which the decision-maker cannot see the whole situation — like a poker player who cannot see other hands — and must act on clues and memory.
particle filterA way to track something uncertain, such as a robot's location, by keeping thousands of random guesses, scoring each against what the sensors see, and copying the best ones.
particle swarm optimizationAn optimization method inspired by flocking birds: candidate solutions 'fly' through the space of possibilities, each pulled toward the best spot it has found and the best spot its neighbors have found.
Pascal's muggingA puzzle about decisions: if someone promises an astronomically huge reward or threatens a huge disaster, even a tiny chance they are honest seems to justify paying them, which looks absurd.
pass@kA score for code-writing AI: the chance that at least one of k attempts at a problem is correct, as checked by actually running tests on the code.
Judea PearlAn Israeli-American computer scientist who gave AI the mathematics for reasoning under uncertainty and, later, for telling cause and effect apart from mere correlation.
Roger PenroseA British mathematical physicist and Nobel laureate who argues that human understanding is not computational, so that no computer, however powerful, could truly think like a mathematician.
Perceptrons (book)A 1969 book by Marvin Minsky and Seymour Papert proving that simple one-layer neural networks cannot learn certain patterns. It is widely blamed for a decade-long slump in neural-network research.
perplexityA score for how surprised a language model is by real text; lower is better. A perplexity of 10 means guessing each next word is, on average, as hard for the model as guessing a ten-sided die's roll.
physics-informed neural networkA neural network that is penalized during training whenever its output breaks a known physical law, such as an equation of fluid flow, so it can model a system well even from few measurements.
PID controllerThe most common automatic controller in industry. It corrects a machine using three ingredients: how far off it is now, how long it has been off, and how fast the error is changing.
Walter PittsA largely self-taught American logician who, at about 20 and without a degree, co-invented with Warren McCulloch a simple logical model of a brain cell, the starting point of neural networks.
policy iterationA method for finding the best strategy in a fully known problem by alternating two steps: work out how good the current strategy is, then switch to better choices wherever possible, until nothing improves.
polynomial regressionFitting a curve instead of a straight line by also using the square, cube and higher powers of an input. Higher powers let the curve bend more but easily start chasing noise.
power lawA rule linking two quantities in which scaling one up by a given factor always changes the other by the same multiple, wherever you start. Plotted on logarithmic axes, it is a straight line.
precisionOf all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder.
predictionA model's output for a given input: its best guess at an unknown answer. In AI it need not be about the future; labeling a photo 'cat' counts as a prediction.
predictive processingThe idea that the brain is a prediction machine: it constantly guesses what its senses will report next and learns mainly from its mistakes, so what we perceive is largely its best guess.
principal component analysisA technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.
privacy budgetThe total amount of privacy risk a data holder agrees to spend; each analysis or release uses up part of it, and once it is gone, no more answers can be given safely.
privileged basisA property of a layer in a neural network where the individual numbered slots, not just blends of them, carry special meaning, because something in the design treats each slot separately.
probabilistic graphical modelA way of describing how many uncertain quantities depend on one another by drawing them as a network, which makes it practical for a computer to reason about all of them together.
probabilistic programmingWriting a statistical model as an ordinary-looking program that includes random choices, and letting the system work out automatically which hidden values best explain the observed data.
probabilistic roadmapA planning method that scatters many random safe robot poses through a space and links nearby ones into a road network, so routes can later be looked up quickly, as on a map.
product quantizationA compression trick for embeddings: chop each long list of numbers into short pieces and replace each piece with the ID of its closest typical piece, shrinking storage many times over while keeping search workable.
projected gradient descent attackA stronger version of the quick pixel-nudging attack: take many small steps that each raise the model's error, pulling the image back after every step so the total change stays tiny.
proper scoring ruleA way of grading probability forecasts that gives the best expected score only to a forecaster who reports what they really believe, so there is nothing to gain by exaggerating or hedging.
propositional logicThe simplest formal logic: whole statements that are either true or false, joined by 'and', 'or', 'not' and 'if… then', with exact rules for working out whether the combination is true.
prover–verifier gameA training setup in which one AI must justify its answers so clearly that a weaker AI can check them, which also makes the strong AI's work easier for people to check.
PutnamBenchA test built from hundreds of problems from the Putnam, North America's top university math contest, written in a form a computer can check, used to measure how well AI systems prove harder mathematics.
Q-functionA score for each possible move in each situation, estimating how well things will turn out, in rewards, if the AI agent makes that move now and sticks to its usual strategy afterward.
QK and OV circuitsA way of splitting what each attention unit in a language model does into two separate questions: where should it look, and what should it pass along from what it sees.
quantile regressionPredicting not just a typical value but a range — say, the delivery time that 90% of orders beat — by fitting models for chosen percentiles of the outcome instead of its average.
quantilizerAn AI decision rule that, instead of picking the single highest-scoring action, picks at random among the top few percent of ordinary, human-like actions — limiting how badly a flawed goal can backfire.
quantizationStoring an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
quantum machine learningResearch into whether quantum computers could learn from data faster or better than ordinary computers, and into using AI to study quantum systems. Practical advantages have not yet been shown.
randomized smoothingA way to make a classifier provably harder to fool: ask it about many noisy copies of an input and take a majority vote, which small deliberate changes cannot easily flip.
rapidly exploring random treeA planning method that grows a branching tree of random moves outward from the robot's starting position, spreading quickly into open space until a branch reaches the goal.
reciprocal rank fusionA simple recipe for merging several ranked lists of search results: each result earns points according to how high it ranks in each list, and the totals decide the final order.
rectified flowA generator trained to move random noise toward real data along straight lines, which can then be straightened further so that a picture comes out in one or a few steps.
regressionPredicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known.
regretHow much worse a learner did than it could have done had it known the best choice from the start — the price of having to learn by trying.
regularizationAny technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
rejection samplingGenerating many candidate answers and keeping only those that pass a check — a scoring model, a test or a correct final answer — often so the AI can then be trained on its own best work.
reparameterization trickA trick that rewrites a random draw as fixed numbers plus separate, outside randomness, so that a neural network can still learn through a step that involves chance.
resolutionA mechanical step of logical reasoning: from 'A or B' and 'not A or C', conclude 'B or C'. Repeating it can demonstrate any conclusion that genuinely follows from what is known.
returnThe total reward an AI agent collects from now until the end of a task, usually counting near-term rewards more than distant ones. Agents are trained to make this total as large as possible.
ridge regressionLinear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs.
robustnessAn AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it.
ROC curveA graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better.
RocqA long-established computer system, known until 2025 as Coq, in which people write mathematical proofs and verified software that the computer checks step by step.
sampling biasA distortion that arises when the data collected over-represents some kinds of people or situations and under-represents others, so conclusions drawn from it don't hold for everyone.
satisfiability modulo theoriesAn extension of logic-puzzle solving that also understands arithmetic, arrays and other data types, so it can check whether conditions in software, such as 'x + y > 10 and x < 3', can ever be met.
satisficingChoosing the first option that is good enough instead of searching for the very best one — a sensible strategy when time, information or brainpower are limited.
scaled dot-product attentionThe standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information.
scaling lawsMeasured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments.
score-based generative modelA generator that learns, for any noisy example, which direction would make it look more like real data — then starts from pure noise and follows those directions step by step.
second-order optimizationTraining methods that use not only the slope of the error but also how that slope is changing — its curvature — to choose better-sized steps. They can need far fewer steps, but each costs more.
secure aggregationA cryptographic trick that lets a server add up updates from many devices while learning only the total, never any single device's contribution.
secure multi-party computationCryptographic methods that let several parties compute a result from their combined private data — say, an average salary — without any of them revealing its own data to the others.
semantic entropyA way to tell when an AI is likely making something up: ask it the same question several times and check whether the answers mean the same thing. Answers that keep changing signal guessing.
sensor fusionCombining readings from several different sensors, such as cameras, radar, lidar, GPS and motion sensors, to get a more accurate and reliable picture than any one sensor gives alone.
ShampooA training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed.
Claude ShannonAn American engineer and mathematician who invented information theory, the mathematics of bits and communication, and wrote one of the first plans for a chess-playing computer.
sharp and flat minimaTwo kinds of low points a trained model can settle into: narrow pits where small changes spoil performance, and broad basins where they barely matter. Broad basins are thought to generalize better, though this is debated.
sigmoid functionAn S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on.
simulated annealingAn optimization method inspired by slowly cooling metal: early on it sometimes accepts worse solutions, to escape dead ends, and becomes pickier as a 'temperature' setting gradually drops.
simultaneous localization and mappingHow a robot, drone or headset builds a map of an unfamiliar place while at the same time working out where it is on that map, using its cameras or laser scanners.
singular learning theoryA mathematical theory of learning for models like neural networks, whose many redundant settings break standard statistics; it explains how such models generalize and how their learning unfolds.
soft computingA group of computing methods, including fuzzy logic, neural networks and evolutionary algorithms, that accept imprecise, approximate answers in exchange for being practical on messy real-world problems.
softmaxA formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.
Ray SolomonoffAn American mathematician who worked out an ideal mathematical recipe for predicting what comes next from past data, and who attended the 1956 meeting that launched AI.
source separationPulling apart a mixture of sounds into its separate parts, such as isolating one voice in a noisy party or the vocals, drums and bass in a finished song.
sparsityThe property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation.
spectral clusteringA clustering method that treats data points as a network linked by similarity and finds natural groups by cutting the network where its links are weakest.
spectrogramA picture of sound that shows time from left to right, pitch from bottom to top, and loudness as brightness or color, so that speech and music can be seen as patterns.
stateA snapshot of the situation an AI agent is in — for a chess program, the arrangement of pieces and whose turn it is — containing what it needs to decide what to do next.
state space modelA type of sequence model that reads text like a running tally, updating a fixed-size internal summary at each step, so its memory and cost do not balloon as texts get longer.
stochastic gradient descentA faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset.
support vector machineA classic method that separates two groups of examples with the widest possible gap — like laying the broadest possible road between them. A mathematical shortcut lets it draw curved boundaries too.
surrogate modelA simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply.
survival analysisMethods for predicting how long until something happens — a patient's relapse, a machine's failure, a customer's cancellation — that properly use the cases where it has not happened yet.
symbolic regressionGetting a computer to discover a readable mathematical formula, such as a law of physics, that fits a set of measurements, instead of only fitting numbers into a formula chosen in advance.
tabu searchA search method that improves an answer by small changes while keeping a short memory of recent moves and forbidding their reversal for a while, so it does not circle back to the same dead ends.
tanhAn S-shaped curve, like the sigmoid but centered on zero, that squeezes any number into a value between −1 and 1. It was the standard neuron rule before ReLU took over.
temperatureA setting that controls how adventurous an AI model's word choices are. Low temperature makes it pick the safest, most likely words; high temperature makes it more varied and surprising, but more error-prone.
temporal logicA logic for statements about time — 'eventually the light turns green', 'the door is never open while the train moves' — used to specify and check how systems must behave over time.
TF-IDFA simple score for picking out the words that characterize a document: a word counts for more the more often it appears there, and for less the more of the collection's other documents also use it.
Thompson samplingA way to choose among uncertain options by picking each one about as often as it is likely to be the best, based on everything learned so far.
time series forecastingPredicting future values of something measured over time, such as sales, electricity demand, prices or rainfall, from its past values and related information.
trajectory optimizationCalculating the best way for a robot to move over time, choosing exact positions, speeds and forces moment by moment so the motion is smooth, fast or energy-efficient and avoids obstacles.
traveling salesman problemThe puzzle of planning a round trip through a list of cities that stops at each one once and covers the least distance. It is easy to state, but the number of possible routes explodes as cities are added.
Alan TuringA British mathematician who worked out what computers can do in principle, helped break German codes in the Second World War, and proposed the best-known test of machine intelligence.
universal approximation theoremA mathematical result showing that a neural network with enough neurons can imitate essentially any smooth input-to-output relationship as closely as you like, though it does not say how to find that network.
universal intelligenceA mathematical definition of intelligence as an agent's average ability to achieve goals across every possible environment, with simpler environments counting more.
updateless decision theoryA theory of rational choice in which an agent commits to the best overall plan for every situation it might face, instead of re-deciding each time it learns something new.
uplift modelingPredicting not who will buy, recover or respond, but whose behavior will actually change because of an action such as a discount or a treatment, so that it is offered only where it makes a difference.
upper confidence boundA rule for choosing among uncertain options that gives each one a bonus for how little is known about it, so promising but rarely tried options get a fair chance.
utility indifferenceA design trick that makes an AI exactly indifferent to whether its off switch is pressed, so that it has no reason either to fight or to trigger shutdown.
Leslie ValiantA British computer scientist who in 1984 gave the first rigorous definition of what it means for a computer to learn from examples, founding the theory of machine learning.
value functionAn estimate of how promising a situation is for an AI agent, judged by the rewards it is likely to collect from that point on, the way a chess player sizes up a board.
value iterationA step-by-step method for finding the best strategy when all the rules and rewards of a problem are known: repeatedly re-score every situation from the scores of the situations it can lead to, until the scores stop changing.
Vladimir VapnikA Soviet-born American mathematician who built the statistical theory of how machines learn from examples and co-invented support vector machines, a leading method before deep learning.
variational autoencoderA neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples.
vector searchSearching by closeness instead of exact match: the query and all stored items are lists of numbers, and the search returns the items whose numbers are most similar to the query's.
vector space modelAn early, still-influential idea for search: treat every document and query as an arrow in a space with one direction per word, and rank documents by how closely their arrows point the same way as the query's.
Viterbi algorithmA method that finds the likeliest hidden sequence behind some observations — such as each word's grammatical role in a sentence — by keeping only the best path to each point instead of trying every combination.
John von NeumannA Hungarian-American mathematician who shaped the design of modern computers, co-founded game theory, and is the first person known to have spoken of a coming technological 'singularity'.
Wasserstein GANA version of the forger-versus-detective image generator that measures how far its fakes are from real data in a smoother way, making training steadier and easier to monitor.
Lotfi ZadehAn Azerbaijani-born American engineer who invented fuzzy logic, which lets computers reason with vague categories such as 'warm' or 'tall' that are true only to a degree.
zero moment pointThe spot on the ground beneath a walking robot where the floor's push balances out; keeping it inside the area under the robot's feet stops the robot from tipping over.