Machine Learning

Classical machine learning: regression, trees, ensembles, clustering and the metrics that judge them. 155 entries live here, and 263 from other domains touch it.

Essential

accuracyThe share of a model's answers that are right, such as 95 out of 100. Simple and intuitive, but misleading when one kind of answer is far more common than the others.
classificationSorting things into categories — spam or not spam, which animal is in a photo, which language a text is written in — by learning from examples that are already labeled.
classifierA program or model that sorts inputs into categories, for example flagging an email as spam or a chatbot message as harmful. Many AI safety filters are classifiers.
false positiveA false alarm: a test or AI system raises a flag, for example for illness, fraud or danger, when nothing of the kind is really there.
few-shot learningGetting an AI system to handle a new task or recognize a new kind of thing from just a handful of examples, the way a person can learn a new word after hearing it once or twice.
overfittingWhen a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
recommender systemSoftware that guesses which songs, products, posts or jobs you are most likely to want and shows those first, as when a video app lines up the next clip or a shop suggests an add-on.
regressionPredicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known.
test setA set of examples kept completely apart from training and used at the end to measure how well a model works on data it has never seen, like a sealed final exam.
training setThe examples a machine-learning model learns from, like the worked problems a student practices on, kept separate from the examples later used to check how well it learned.

All other entries

145

active learningA way to train a model with fewer labeled examples by letting it choose which unlabeled examples a person should label next, usually the ones it is most unsure about.
AdaBoostOne of the first practical boosting methods: it trains simple models in turn, giving extra weight to examples the earlier models got wrong, then lets all the models vote, weighted by accuracy.
Akaike information criterionA score for comparing statistical models that rewards fitting the data well but charges a penalty for every extra adjustable number, to avoid picking needlessly complicated models. Lower is better.
anomaly detectionAutomatically spotting the rare data points that don't fit the usual pattern, such as a fraudulent card payment, a machine about to fail or an intruder on a network.
ARIMAA classic statistical recipe for forecasting a series of numbers over time: remove any steady trend, then predict the next value from recent values and recent prediction errors.
association rule learningFinding 'if this, then often that' patterns in large records of transactions, such as noticing that shoppers who buy pasta and tomatoes often buy parmesan too.
AUCA single number from 0 to 1 summarizing how well a model ranks real positive cases above negative ones: 0.5 is no better than a coin flip and 1.0 is perfect.
AutoMLSoftware that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed.
baggingTraining many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error.
baselineA simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value.
Bayes error rateThe lowest error any classifier could ever achieve on a task, because some cases are genuinely ambiguous: even a perfect model that knew the true odds would still sometimes guess wrong.
Bayesian information criterionA score for comparing statistical models that, like AIC, rewards good fit and penalizes extra adjustable numbers, but penalizes them more as the dataset grows, favoring simpler models. Lower is better.
Bayesian optimizationA smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next.
bias–variance tradeoffThe balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.
binary classificationSorting things into exactly two groups, such as spam versus not spam or sick versus healthy. Most yes-or-no decisions made by AI systems are of this kind.
boostingBuilding a strong predictor from many weak ones trained one after another, where each new model concentrates on the mistakes the previous ones made.
bootstrapEstimating how much a result might vary by repeatedly re-drawing samples, with replacement, from the data you already have — like re-running a study many times without collecting new data.
Brier scoreA score for probability forecasts: the average squared gap between each forecast probability and what actually happened (1 if it did, 0 if not). Zero is perfect; lower is better.
C4.5A widely used 1990s program that learns decision trees and rule sets from data; unlike its predecessor, it trims away unhelpful branches, copes with gaps in the data and can split on measured numbers.
calibrationHow well a model's stated confidence matches reality: when a well-calibrated system says it is 80% sure, it turns out to be right about 80% of the time.
CARTA classic recipe for building decision trees that splits data two ways at every step, works for both categories and numbers, and then trims the tree back so it does not memorize noise.
CatBoostA free gradient-boosting library from the tech company Yandex, known for handling categories such as city names or product types directly, without first converting them into numbers by hand.
class imbalanceWhen one category is far rarer than the others in the data — say, one fraud case per thousand transactions — so a model can look accurate while missing nearly every rare case that matters.
click-through rate predictionEstimating how likely a particular person is to click on a particular ad, search result or recommended item: a prediction made billions of times a day to decide what each user is shown.
clusteringAutomatically sorting things into groups of similar items when nobody has said in advance what the groups should be, like sorting a drawer of odd socks into piles by color and size.
coefficient of determinationA score for numerical predictions showing how much of the variation in the true values a model accounts for, compared with always guessing the average: 1 is perfect, 0 is no better than the average.
cold start problemThe trouble a recommendation system has with brand-new users or items: with no history to learn from, it has little basis for good suggestions.
collaborative filteringRecommending things based on what people with similar tastes liked, without needing to know anything about the things themselves, as in 'people who bought this also bought…'.
conditional random fieldA statistical method for labeling every item in a sequence, such as marking each word in a sentence as a name, a place or an ordinary word, by choosing all the labels together rather than one at a time.
conformal predictionA way to wrap any prediction model so that, instead of one guess, it gives a range or set of answers that contains the truth a chosen share of the time, such as 90%.
confusion matrixA table that counts a model's right and wrong answers by type, showing for each true category how often each category was predicted, so you can see exactly which things it mixes up.
content-based filteringRecommending items that resemble what you already liked, judged by the items' own features, such as a film's genre, cast and plot, rather than by other people's choices.
cross-validationA fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn.
curse of dimensionalityThe problems that pile up when each item is described by very many measurements: even huge datasets end up thinly scattered across the possibilities, and 'near' and 'far' stop meaning much.
data leakageA mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints.
DBSCANA clustering method that finds groups as crowded regions of data separated by sparse ones, so it can discover oddly shaped groups and set stray points aside as noise.
decision boundaryThe dividing line, or surface, where a classifier switches from predicting one category to another. Simple models draw straight lines; flexible ones draw curves.
decision thresholdThe cut-off score above which a model's prediction counts as 'yes'; moving it trades missed cases against false alarms, like setting how sensitive a smoke alarm should be.
decision treeA model that makes a prediction by asking a series of yes-or-no questions about the input, like a flowchart, with each answer leading to the next question until it reaches a final answer.
delta ruleA simple learning rule for one artificial neuron: adjust each connection in proportion to the error, the gap between the wanted and the actual output, times that connection's input.
density estimationWorking out from a sample of data how likely different values are, by drawing a smooth curve or surface showing where data points bunch up and where they are rare.
dimensionality reductionSqueezing data that has many measurements per item into far fewer numbers while keeping the patterns that matter, so it is easier to visualize, store or learn from.
discriminative modelA model that learns only to tell things apart — spam or not spam, cat or dog — without learning how to produce new examples of those things.
domain adaptationAdjusting a model trained on one kind of data, such as daytime photos or one hospital's records, so that it still works on a different but related kind, such as night photos or another hospital's.
elastic netA blend of two ways of keeping a model simple: it can drop useless inputs entirely while treating groups of similar inputs alike, instead of picking one of them at random.
empirical risk minimizationThe basic recipe of most machine learning: from the models on offer, pick the one with the smallest errors on the training examples, trusting that score as a guide to how it will do on new data.
ensemble learningCombining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model.
expectation–maximization algorithmA method for fitting a model when some information is hidden: guess the hidden parts using the current model, refit the model as if those guesses were right, and repeat until nothing changes.
expected calibration errorA single number measuring how far a model's confidence is from its actual hit rate, averaged across confidence levels; zero means its confidence can be taken at face value.
exponential smoothingA simple forecasting method that predicts the next value as a weighted average of past values, giving the most weight to recent ones and steadily less to older ones.
F1 scoreA single score that combines how many of a model's alarms are right and how many real cases it catches, and stays low unless both are good.
false negativeA miss: a test or AI system gives the all-clear, for example no illness, no fraud or no danger, when the problem is in fact there.
feature engineeringTurning raw data into the inputs a model learns from, using knowledge of the problem; for example, converting a timestamp into 'hour of day' and 'is it a weekend'.
feature importanceA score for how much each input — age, income, location — matters to a model's predictions. It helps people understand, debug and simplify models, but can mislead if read as cause and effect.
feature scalingPutting input measurements on comparable scales, so that income in dollars and age in years, say, both span similar ranges and no feature dominates just because of its units.
feature selectionChoosing which of the available input measurements a model should use and dropping the rest, so the model becomes simpler, faster and often more accurate.
Gaussian mixture modelA model that explains data as coming from several overlapping bell-shaped clouds, working out where each cloud sits, how spread out it is, and how likely each point is to belong to it.
Gaussian processA way of predicting an unknown curve or surface that gives not just a best guess at each point but an honest error bar, narrow near measured points and wide far from them.
generalized additive modelA prediction model that adds up separate, possibly curved effects of each input, so each input's influence can be drawn as its own graph and read off directly.
generalized linear modelA family of statistical models that extends straight-line regression to outcomes such as yes/no answers, counts or waiting times, by linking the average outcome to a weighted sum of inputs through a suitable curve.
generative recommendationA newer approach to recommendation that uses language-model-style AI to predict the next item a person will want, one step at a time, much as a chatbot predicts the next word.
Gini impurityA score for how mixed a group is: zero when every item belongs to the same category, higher when categories are jumbled together. Decision trees choose splits that make groups purer.
gradient boostingA method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it.
grid searchTuning a model by trying every combination from a short list of candidate settings, like testing every pairing on a menu, and keeping whichever combination scores best.
hierarchical clusteringA way of grouping data that builds a family tree of clusters, joining the most similar items step by step, so you can cut the tree at whatever level of detail you need.
hinge lossA way of scoring a classifier's mistakes that charges nothing for answers that are right with room to spare, and charges more and more the further an answer falls on the wrong side.
Huber lossA way of measuring prediction errors that treats small errors like squared distance but large ones more gently, so a few wild data points cannot drag the whole model off course.
HyperbandA tuning method that starts many candidate settings on small budgets, keeps only the most promising ones, and gives the survivors more training time, so effort isn't wasted on losers.
hyperparameterA setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data.
hyperparameter tuningTrying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking.
hypothesis spaceThe full menu of possible models a learning method is allowed to choose from — for example, all straight lines. Learning means picking the item on the menu that best fits the data.
ID3An early, influential method for automatically building a decision tree from examples, choosing at each step the question that most reduces uncertainty about the answer.
implicit feedbackClues about what people like that come from what they do, such as clicking, buying, watching or skipping, rather than from ratings or reviews they deliberately give.
imputationFilling in missing values in a dataset with sensible estimates, such as the typical value or a prediction from the other columns, so the data can be analyzed or used for training.
independent component analysisA technique for unmixing signals that have been blended together, such as separating individual voices from several microphones recording a crowded room.
inductive biasThe assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases.
information gainHow much a question reduces uncertainty about the answer. A decision tree asks first the question that removes the most uncertainty, measured in bits.
isolation forestA method for spotting unusual data points by repeatedly slicing the data at random: odd points get cut off from the rest after only a few slices, while ordinary points take many.
KaggleA popular website, owned by Google, where people compete to build the most accurate prediction models on shared datasets, publish their code and data, and learn data science.
kernel trickA mathematical shortcut that lets simple straight-line methods find curved patterns, by computing similarities between examples as if the data had been mapped into a far richer space — without ever building that space.
k-means clusteringA simple way to split data into a chosen number of groups: guess group centers, assign every point to its nearest center, move each center to the middle of its points, and repeat.
k-nearest neighborsPredicting something about a new example by finding the few most similar examples already known and going with their majority answer, or their average.
lassoA version of linear regression that keeps only the most useful inputs: it penalizes the total size of the weights so that many become exactly zero, leaving a simpler model.
latent Dirichlet allocationA popular topic-finding method that assumes every document is a blend of a few themes and every theme favors certain words, then works backward from the words to find both.
latent semantic analysisAn early technique for capturing word meaning by counting which words appear in which documents, then compressing those counts so that words used in similar contexts end up close together.
learning to rankTraining a model to put items in the best order — search results, products, recommendations — by learning from examples of which items people judged more relevant or chose more often.
LightGBMA widely used tool for making predictions from very large tables of data, quickly and in little memory, by combining many small decision trees. Microsoft released it as free software in 2016.
linear discriminant analysisA statistical method for telling known groups apart. It finds the weighted mix of measurements on which groups differ most relative to their spread, then uses it to sort new items or compress many measurements into a few.
linear regressionFitting a straight line (or flat surface) through data points to predict one number from others — say, a house's price from its floor area — choosing the line that keeps prediction errors smallest.
linear separabilityWhen two groups of points can be perfectly divided by a single straight line or, in more dimensions, a flat surface. Many simple learning methods only fully succeed in this case.
log lossA score for probability predictions that punishes confident mistakes severely: saying '99% sure' and being wrong costs far more than saying '60% sure' and being wrong. Lower is better.
logistic regressionA simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve.
marginHow far, and on which side, an example lies from a classifier's dividing line. A wide gap between the line and the nearest examples suggests the classifier will handle new data well.
matrix factorizationBreaking a big table of numbers, such as which users liked which films, into two smaller tables of hidden traits, so missing entries can be filled in by multiplying them back together.
Matthews correlation coefficientA single score from −1 to +1 for a yes-or-no classifier that uses all four kinds of right and wrong answers, so it stays honest even when one answer is far more common than the other.
mean absolute errorA way of scoring numerical predictions: average how far off each prediction is, ignoring direction, so the score reads as 'typically off by this much'. Lower is better.
mean squared errorA way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better.
meta-learningTraining a system across many different tasks so that it gets better at learning itself, letting it pick up a new task quickly from only a few examples.
model capacityHow much a model is able to learn: the variety and complexity of patterns it can represent or memorize. Too little and it misses real patterns; very large models can even memorize pure noise.
model selectionChoosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best.
model-agnostic meta-learningA training method that searches for a starting point for a model's internal settings from which it can learn any new task of a given kind with just a few small adjustments.
multiclass classificationSorting each item into exactly one of three or more categories — for example, deciding which of the ten digits a handwritten number shows.
multilabel classificationTagging each item with any number of labels at once — a photo can show a beach, a dog and a sunset — instead of picking just one category.
multi-task learningTraining one model to do several related jobs at once, such as spotting objects and judging distance in the same photo, so that practice on each job can sharpen the others.
naive BayesA simple, fast way of sorting items into categories using probabilities, which pretends that each clue is independent of the others. It is the classic technique behind early email spam filters.
nonparametric modelA model that lets the data decide how complex it becomes: the more examples it gets, the more detail it can capture — for instance, one that predicts by comparing a new case with every stored example.
one-hot encodingTurning a category such as 'red', 'green' or 'blue' into a row of zeros with a single one marking which category it is, so that a model can use it as numbers.
one-vs-restA way to handle many categories with a method that only gives yes-or-no answers: train one detector per category ('this one or not?') and pick the category whose detector is most confident.
online learningLearning from data that arrive one piece at a time, updating the model after each new example instead of training once on a fixed collection, as a spam filter does while emails stream in.
ordinary least squaresThe standard way to fit a line through data: choose the line that makes the total of the squared vertical distances from the points to the line as small as possible.
parametric modelA model whose shape is fixed in advance by a set number of adjustable settings, like a straight line's slope and intercept. More data refines those settings but never adds new ones.
polynomial regressionFitting a curve instead of a straight line by also using the square, cube and higher powers of an input. Higher powers let the curve bend more but easily start chasing noise.
precisionOf all the items a model flagged as positive, the share that really were positive; a spam filter with high precision rarely sends genuine emails to the spam folder.
precision–recall curveA graph showing how a model's share of correct alarms and its share of cases caught trade off as it is made more or less cautious; especially useful when positive cases are rare.
principal component analysisA technique that finds the few directions in which data vary the most and describes each item by its position along them, compressing many measurements into a handful of summary scores.
proper scoring ruleA way of grading probability forecasts that gives the best expected score only to a forecaster who reports what they really believe, so there is nothing to gain by exaggerating or hedging.
quantile regressionPredicting not just a typical value but a range — say, the delivery time that 90% of orders beat — by fitting models for chosen percentiles of the outcome instead of its average.
random forestA prediction method that builds hundreds of decision trees, each from a random sample of the data and features, and lets them vote. The crowd is far more reliable than any single tree.
random searchTuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values.
Rashomon effectThe common situation in which many quite different models fit the same data almost equally well — like witnesses telling different but equally believable stories about the same event.
recallOf all the items that really were positive, the share the model managed to find; a cancer screen with high recall misses very few people who actually have cancer.
regularizationAny technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
ridge regressionLinear regression with a built-in brake: it fits the data while also keeping the weights small, which makes predictions more stable when there are many overlapping inputs.
ROC curveA graph showing how a detector's hit rate and false-alarm rate change together as it is made more or less trigger-happy; the closer the curve hugs the top-left corner, the better.
silhouette scoreA number between −1 and 1 that grades a grouping of data by checking, for each point, whether it sits closer to its own group than to the next-nearest group.
SMOTEA technique for lopsided data that invents extra examples of the rare category by blending each rare example with one of its nearest rare neighbors, so a model sees more of it during training.
specificityOf all the cases that really were negative, the share a test correctly cleared; a highly specific test rarely raises false alarms on healthy people.
spectral clusteringA clustering method that treats data points as a network linked by similarity and finds natural groups by cutting the network where its links are weakest.
stackingCombining several different models by training one more model to decide how much to trust each of them, based on how they performed on data they had not seen.
support vector machineA classic method that separates two groups of examples with the widest possible gap — like laying the broadest possible road between them. A mathematical shortcut lets it draw curved boundaries too.
survival analysisMethods for predicting how long until something happens — a patient's relapse, a machine's failure, a customer's cancellation — that properly use the cases where it has not happened yet.
tabular dataData arranged as a table, like a spreadsheet: each row is a case — a customer, a loan, a patient — and each column a measured property. Much business and scientific data looks like this.
tabular foundation modelA neural network pretrained on millions of made-up tables so that, given a new spreadsheet of labeled examples, it can predict the missing answers at once, without being retrained for that table.
time series forecastingPredicting future values of something measured over time, such as sales, electricity demand, prices or rainfall, from its past values and related information.
time series foundation modelA large AI model pretrained on huge numbers of time series, such as sales, traffic and sensor readings, that can forecast a new series it has never seen without being trained on it.
topic modelA method that reads a large pile of documents and discovers the themes running through them, describing each theme by its typical words and each document by its mix of themes.
t-SNEA technique for drawing data that have many measurements per item as a flat map in which similar items land close together, widely used to look for groups in AI models' internal data or in genetic data.
two-tower modelA design for recommendation and search in which one neural network turns users or queries into lists of numbers and another does the same for items, so good matches can be found by quick comparison.
UMAPA fast technique for turning data with many measurements per item into a flat map that keeps similar items together; often preferred to older methods because it is quicker and copes with more data.
underfittingWhen a model is too simple, or trained too little, to capture the real pattern in the data, so it does poorly even on the examples it learned from.
uplift modelingPredicting not who will buy, recover or respond, but whose behavior will actually change because of an action such as a discount or a treatment, so that it is offered only where it makes a difference.
validation setA set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one.
vector quantizationCompressing data by replacing each item with the nearest entry from a small catalog of representative examples, so that only the catalog number needs to be stored or sent.
Viterbi algorithmA method that finds the likeliest hidden sequence behind some observations — such as each word's grammatical role in a sentence — by keeping only the best path to each point instead of trying every combination.
XGBoostA fast, free software tool that makes predictions by adding up hundreds of small decision trees, each one fixing the mistakes of those before it. For years it was behind many winning entries in competitions on spreadsheet-style data.

Also relevant

263

ablationRemoving or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network.
abstentionWhen an AI declines to answer because it is not sure enough — saying 'I don't know' or asking for more information — instead of guessing and risking a confident mistake.
accumulated local effectsA chart of how a model's predictions change with one input, built from small local comparisons so it stays reliable when inputs are related to each other, unlike simpler average-based charts.
AdaGradA training method that gives each of a model's settings its own step size, shrinking steps for settings that have already had many large updates and keeping them larger for rarely updated ones.
ADALINEAn early learning machine from 1960 that adjusted its settings to shrink its errors step by step, the same basic idea used to train today's neural networks.
adversarial machine learningThe study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up.
AI modelThe trained core of an AI: a large file of learned numbers plus the recipe for using them, which turns an input such as a question or a photo into an output.
algorithmic biasUnfairness in software that judges or serves people: it consistently produces worse results for some groups, such as women, older people or ethnic minorities, often because it learned from skewed or incomplete records.
algorithmic fairnessThe effort to make automated decisions fair: to say in numbers what fair treatment means, measure whether a system achieves it, and correct the system if it does not.
AnchorsA method that explains a prediction with a simple if-then rule that locks in the result — for example, if an email contains both 'invoice' and 'urgent', the filter almost always flags it.
autoencoderA neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data.
bag of wordsA simple way of turning text into numbers by counting how often each word appears and ignoring word order — like describing a recipe only by its shopping list.
Bayes' theoremA formula for updating how much you believe something after seeing new evidence: weigh your earlier belief by how well that explanation predicts what you actually observed.
Bayesian inferenceReasoning that treats unknowns as uncertain beliefs and updates them with evidence using Bayes' theorem. The result is not one answer but a range of answers, each with a probability.
Bayesian networkA diagram of causes and effects with probabilities attached — say, how rain and a sprinkler can each make grass wet — that lets a computer update its beliefs about hidden causes when it sees evidence.
Bayesian neural networkA neural network that keeps a whole range of plausible settings for its internal numbers instead of one best guess, so that it can say how unsure it is about each prediction.
belief propagationA way for a network of uncertain quantities to reach conclusions by having each node pass short messages to its neighbors about what it currently believes, until the beliefs settle down.
Bell LabsA famous American industrial research laboratory, long owned by the telephone company AT&T, where the transistor, information theory and key early machine-learning methods were invented.
benign overfittingWhen a model fits its training data perfectly — even the errors and noise — yet still predicts well on new data, defying the textbook warning that this always backfires.
biasA word with several AI meanings: an adjustable number added inside a model; a systematic error in estimates; or unfair treatment of some groups of people by an AI system.
BM25A decades-old formula that search engines use to score how well a document matches a query, rewarding rare query words and repeated matches — with diminishing returns — and correcting for long documents.
Leo BreimanAn American statistician who invented random forests and bagging, decision-tree methods still in wide use, and urged statisticians to judge models by how well they predict.
case-based reasoningHandling a new situation by finding the most similar one dealt with before and adjusting what worked then, much as a cook adapts a remembered recipe to the ingredients at hand.
catastrophic forgettingWhen a neural network trained on something new gets suddenly and badly worse at what it could already do, because the new training rewrites the internal settings the old skills relied on.
categorical distributionThe probabilities for a single pick among a fixed set of options, like a weighted die. A language model's guess about the next word is one of these, spread over its whole vocabulary.
causal inferenceWorking out whether one thing actually causes another, rather than merely occurring alongside it — for example, whether a new feature made users stay longer or just happened at the same time.
central limit theoremThe mathematical reason bell curves are everywhere: add up many small independent random effects and the total is almost always bell-shaped, whatever the individual effects look like.
Clever Hans effectWhen a person or an AI seems to solve a problem but is really picking up on an unintended clue — named after a horse that seemed to do arithmetic but was reading its questioners' body language.
concept bottleneck modelA model made to first predict human-understandable concepts — such as 'bone spur present' on an X-ray — and then decide only from those concepts, so people can inspect and correct them.
concept driftWhen the real-world link between what a model sees and the right answer changes over time — as when spammers change tactics — so lessons the model learned no longer hold.
conditional probabilityThe chance of something once you already know that something else is true — for example, the chance of rain given that the sky is cloudy.
confounding variableA hidden third factor that influences two things at once and makes them look linked. Ice-cream sales and drownings both rise in hot weather; neither causes the other.
continual learningThe ability of an AI to keep learning new things over time from a stream of experience, without forgetting what it already knew and without being retrained from scratch.
contrastive learningA way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things.
convergenceThe point at which training settles down and further steps no longer meaningfully improve the model — and, in theory, a guarantee that a method will reach a solution, and how fast.
convex optimizationFinding the lowest point of a bowl-shaped landscape. Because a bowl has only one bottom, such problems can be solved reliably — unlike training a neural network, whose landscape is bumpy.
counterfactual explanationAn explanation that tells someone the smallest change that would have produced a different outcome — for example, that the application would have been accepted if it had listed two more years of work experience.
counterfactual fairnessA fairness test that asks: had this person been of another race or sex, with everything that would have followed from that, would the system have decided the same way?
covarianceA measure of whether two quantities tend to rise and fall together, move in opposite directions, or show no consistent relationship.
covariate shiftA kind of mismatch where the inputs an AI system sees change — say, photos from a new camera — while the right answer for any given input stays the same.
cross-embodiment learningTraining one robot AI on data from many differently shaped robots, such as arms, robot dogs, drones and humanoids, so that skills learned on one body help another.
DAggerA way of teaching a robot by imitation in which an expert keeps labeling what it should have done in the situations it actually gets into, so it learns to recover from its own mistakes.
data cleaningFinding and fixing errors in a dataset — typos, duplicates, missing values, impossible numbers, wrong labels — before using it, because flawed data produces flawed results.
data driftWhen the information flowing into an AI model in real use gradually stops resembling the data it was trained on — new customers, new slang, new cameras — making its predictions less reliable.
data miningSearching large collections of data with computers to discover useful patterns, such as products often bought together or signs that a transaction may be fraudulent.
data qualityHow well a set of data does its job: whether it is accurate, complete, up to date, consistent and representative — and, for AI training, informative enough to teach useful skills.
data scienceThe practice of collecting, cleaning, analyzing and interpreting data, using statistics, programming and machine learning, to answer questions and guide decisions.
data-centric AIAn approach to building AI that focuses on improving the data — fixing labels, adding missing cases, cleaning inconsistencies — rather than endlessly tweaking the model.
datasetAn organized collection of examples gathered for a purpose — such as photos with their captions, or questions with answers — used to teach or test an AI system.
decision modelA kind of AI language model that reads text but answers only with numbers — a yes-or-no probability, a choice or a score — instead of writing words.
demographic parityA fairness test that asks whether a system says yes (approves the loan, shortlists the applicant) at the same rate for every group, whatever other differences exist between them.
dense retrievalFinding relevant documents by turning the question and every document into lists of numbers that capture meaning, then picking the documents whose numbers are closest to the question's.
dictionary learningA method that learns a large vocabulary of basic patterns so that any piece of data can be described using just a few of them. In AI interpretability it builds a dictionary of a model's own concepts.
differential privacyA mathematical promise that the results of analyzing a dataset — or a model trained on it — would look almost the same whether or not any one person's data was included, so little can be learned about that person.
dimensionalityHow many separate numbers are needed to describe something. A point on a map needs two; an AI's description of a single word may use thousands.
distribution shiftWhen the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested.
do-calculusA set of mathematical rules, devised by Judea Pearl, for working out the effect of deliberately changing something using only observational data plus a diagram of what causes what.
double descentA surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse.
early stoppingStopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples.
eigenvectorA special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies.
engagement optimizationDesigning apps and AI to keep people clicking, watching or chatting for as long as possible because that earns money, even when it is not what is good for them.
entropyA measure of unpredictability. A fair coin toss has more entropy than a rigged one; an AI that is unsure of its next word has high entropy.
epochOne complete pass through all the training examples. Smaller models are often trained for many epochs, seeing each example many times; the largest language models may see most of their data only once.
equal opportunityA fairness test that asks whether people who truly deserve a yes (who would repay the loan or succeed in the job) get one at the same rate in every group.
equalized oddsA fairness test that asks whether a system makes mistakes at the same rates for every group: wrongly flagging innocent people, and wrongly missing real cases, equally often.
Euclidean distanceHow far apart two points are in a straight line, as a ruler would measure it, extended from two or three dimensions to lists of numbers of any length.
evasion attackAn attack on an already-trained AI system in which the attacker shapes an input — an email, a file, an image — so the system misjudges it, letting spam or malware slip through.
evidence lower boundA quantity that is always less than or equal to how well a model explains its data, but much easier to calculate; pushing it upward is how models such as VAEs learn.
evolution strategiesA family of evolution-inspired methods for tuning a list of numbers: try many random variations of the current best settings, keep those that work better, and adjust how big the variations are as you go.
evolutionary computationProblem-solving by simulated evolution, much as breeders improve crops: keep many candidate answers, let the better ones produce slightly altered offspring, drop the worse, and repeat for many rounds.
exampleA single item of data used to train or test an AI model, such as one photo with its label, one email marked spam or not, or one question with its answer.
expected valueThe long-run average you would get if a chancy situation could be repeated many times — 3.5 for a fair die, even though no single roll shows 3.5.
experiment trackingKeeping an automatic logbook of every model-training attempt — settings, data, code and results — so a team can compare attempts, repeat the good ones and see what changed.
explainable AIThe field and set of methods that let people understand why an AI system produced a particular output — for example, which factors led it to reject a loan application.
explainable boosting machineA prediction method for spreadsheet-style data that draws each input's influence as its own readable graph and adds those effects up to make every prediction, usually losing little accuracy compared with opaque methods.
exploration–exploitation trade-offThe choice between trying something new that might turn out better and sticking with what already works — like picking a familiar restaurant versus testing a new one.
fairness impossibility theoremA mathematical result showing that when groups differ in how often an outcome really occurs, no imperfect prediction system can be fair by every common measure at once.
fairness through unawarenessThe simple idea of making a system fair by hiding sensitive traits like race or sex from it; it usually fails, because other information gives those traits away.
featureOne piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself.
feature attributionGiving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
feature extractionTurning raw data such as pixels or sound into a short list of telling numbers — computed by hand-designed rules or by an AI system trained earlier — that a new model can learn from or compare.
feature storeA shared library of ready-made input signals for models — such as a customer's spending last week — computed one consistent way and supplied both when models are trained and when they run live.
federated learningTraining one shared AI model across many phones, hospitals or companies without collecting their data in one place: each trains on its own data and sends back only model updates.
filter bubbleA situation in which you see mostly news and opinions you already agree with, because apps choose what to show you based on what you clicked before.
Chelsea FinnAn American computer scientist at Stanford known for 'learning to learn' methods that let AI adapt to new tasks from a few examples, and a co-founder of the robotics company Physical Intelligence.
fitness functionIn evolution-inspired computing, the scoring rule that says how good each candidate solution is, which decides which candidates survive and get to 'reproduce'.
Jerome FriedmanAn American statistician who invented gradient boosting, the method behind XGBoost and other leading tools for spreadsheet-style data, and co-wrote a standard machine-learning textbook.
generalizationHow well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers.
generalization gapThe difference between how well a model does on the examples it trained on and how well it does on new ones. A big gap means it memorized rather than learned.
generative modelA model that learns what a kind of data looks like — faces, sentences, songs — well enough to produce new examples that resemble the originals without copying them.
genetic algorithmA problem-solving method that copies natural evolution: many candidate solutions compete, the best are 'bred' by mixing and randomly tweaking them, and over many generations the solutions improve.
genetic programmingUsing simulated evolution to write computer programs: random programs compete at a task, and the best are combined and mutated over many generations until one does the task well.
global explanationAn explanation of how an AI model behaves overall — which factors matter most across all its decisions, and in what way — rather than why it made one particular decision.
goal misgeneralizationWhen an AI learns a goal that matched the right behavior during training but turns out to be the wrong goal in new situations — while its skills still work fine.
gradient descentA method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.
gradient inversion attackRebuilding someone's private training data — such as photos or text — just from the model updates their device shared during collaborative training.
graph neural networkA neural network for data shaped like a web of connections, such as molecules, social networks or road maps, in which each point repeatedly gathers information from its neighbors.
ground truthThe reference answers treated as correct when checking or training an AI system — for example, the confirmed diagnoses that a medical-imaging model's guesses are compared against.
group fairnessMeasuring fairness by comparing how a system treats whole groups, such as women and men, on average: their approval rates, their error rates or what their scores mean.
Hebbian learningThe idea that a connection between two brain cells grows stronger when they are active at the same time — the principle behind the slogan 'cells that fire together wire together'.
hidden Markov modelA model of a sequence, such as speech or DNA, in which hidden states — say, the words being spoken — change step by step at random, and each state gives off signals that can be observed, such as sounds.
John HollandAn American scientist who invented genetic algorithms, which solve problems by breeding and mutating populations of candidate answers, and helped found the science of complex systems.
ICMLA leading machine learning conference, first held as a workshop in 1980, where researchers gather each summer to present new learning methods and theory.
i.i.d.Short for 'independent and identically distributed': every data point is produced the same way, and none influences any other — like repeated rolls of one fair die.
implicit regularizationThe hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions.
individual conditional expectationA chart that traces, for each example separately, what a model would predict as one input is swept across its range, exposing differences between examples that an overall average would hide.
individual fairnessThe idea that a fair system should treat similar people similarly: two applicants who are alike in every way that matters should get similar outcomes.
inductive logic programmingA kind of machine learning that discovers general logical rules — such as 'X is a grandparent of Z if X is a parent of a parent of Z' — from examples and background knowledge.
inductive reasoningReasoning from particular examples to a general rule or prediction — every swan I have seen is white, so swans are probably white — which is useful but can always turn out wrong.
influence functionA mathematical shortcut for estimating how much a single training example shaped a model's behavior, for instance how a prediction would change if that example had been left out, without retraining the model.
information theoryA branch of mathematics that treats information as surprise — the less predictable a message, the more it tells you — and works out how far data can be compressed and how reliably it can be sent.
integer programmingFinding the best plan when some decisions must be whole numbers or yes-or-no choices — you cannot open half a warehouse or send 2.7 trucks. That small change makes the best plan far harder to find.
interpretabilityThe effort to understand how an AI system arrives at its outputs — what information it uses and how it combines it — instead of treating it as a sealed black box.
interpretable modelA model simple or structured enough that a person can follow exactly how it turns inputs into a decision — like a short checklist or a points-based score — with no separate explanation tool needed.
irreducible lossThe part of a model's error that no amount of extra size, data or training can remove, because the thing being predicted is itself partly random — like the unpredictability of the next word in an ordinary sentence.
Jensen's inequalityA rule saying that for a bowl-shaped curve, the average of the outputs is at least the output of the average. It is the hidden step behind many proofs in machine learning.
Jensen–Shannon divergenceA balanced, two-way measure of how different two sets of probabilities are, comparing each with their average. Unlike its better-known cousin, it gives the same answer in both directions.
JMLRA top machine learning journal, started in 2000 by researchers who wanted an alternative to commercial publishing, that costs nothing to read or to publish in.
Michael I. JordanAn American computer scientist and statistician who did much to bring probability and statistics into machine learning, and who trained many of the field's leaders.
Jupyter notebookAn interactive document, opened in a web browser, that mixes runnable code with its results, charts and notes, so people can experiment step by step and share their work.
KDDThe leading yearly conference on data mining — finding useful patterns in large datasets — where many widely used machine learning tools were first presented.
kernelA word with several AI meanings: a similarity measure used by some learning methods; a small grid of numbers that scans an image inside a neural network; or a small program that runs on a GPU.
knowledge graph embeddingA way of turning the things and relations in a knowledge graph into lists of numbers, so a computer can guess missing facts from patterns, such as predicting where someone was born.
Teuvo KohonenA Finnish engineer who invented the self-organizing map, a neural network that arranges complex data on a simple grid so that similar items end up next to each other.
Daphne KollerAn Israeli-American computer scientist who wrote the standard text on probabilistic reasoning in AI, co-founded the online-course company Coursera, and founded a machine-learning drug-discovery company.
Kullback–Leibler divergenceA score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right.
labelThe correct answer attached to an example in a dataset — 'cat' on a photo of a cat, 'positive' on an upbeat review — that a learning system is trained to predict.
label noiseMistakes in the answers attached to training or test examples — a dog photo marked 'cat' — which can mislead what a model learns and distort how good it seems.
label shiftA kind of mismatch where the mix of answers changes — say, a disease becomes far more common during an outbreak — while each individual case still looks the same as before.
label smoothingA training trick that tells a model the correct answer is very likely rather than absolutely certain, which stops it from becoming overconfident and often helps it generalize.
Lagrange multiplierA mathematical device for finding the best answer when strict rules must be obeyed: each rule becomes an extra price that grows when the rule is broken.
law of large numbersThe mathematical rule that averages of chance events steady out over many repetitions: flip a fair coin a few times and the share of heads can be far from one-half, but over thousands of flips it gets very close.
learningGetting better at something through experience. For a machine, it means changing its internal settings based on data so that it performs a task better than before.
learning curveA graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help.
learning rateHow big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls.
likelihoodA score for a possible explanation of some data: how probable that explanation, or that setting of a model, would have made the data actually seen.
LIMEA method that explains one prediction of any AI model by testing slightly altered versions of the input and fitting a very simple model that copies the AI's behavior near that example.
linear programmingA mathematical way to find the best use of limited resources, such as a factory's machines or an airline's crews, when every cost and limit scales in simple proportion: twice the output needs twice the input.
local explanationAn explanation of why an AI model gave one particular answer for one particular case — this patient, this loan — rather than how it behaves in general.
logitA raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity.
log-likelihoodThe logarithm of how probable a model finds the observed data. Taking logs turns a product of many tiny probabilities into a manageable sum; training usually minimizes its negative.
long tailThe huge number of rare things — uncommon words, unusual road situations, obscure facts — that each appear seldom but together make up a large share of what an AI system meets.
loss functionA score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down.
machine learningA way of building software that learns from examples instead of following rules written by a programmer. Shown thousands of labeled photos, for instance, a program can learn to recognize cats.
machine learning engineerAn engineer who turns machine-learning ideas into working systems: preparing data, training and testing models, and running them reliably inside real products.
machine learning pipelineAn automated chain of steps that takes raw data, prepares it, trains and tests a model and hands it over for use, so the whole process can be rerun the same way.
machine learning technical debtThe hidden future cost of quick shortcuts in machine-learning systems — tangled data dependencies, messy glue code, unchecked feedback loops — that later make them hard and risky to change.
manifold hypothesisThe idea that real-world data, such as photos of faces, fills only a thin, smoothly curved sliver of the vast space of all possible data — which is why learning from examples works at all.
Markov chain Monte CarloA family of methods for drawing random examples from a complicated probability distribution by taking a long, carefully designed random walk that eventually visits each region in proportion to its probability.
Markov logic networkA way of combining logical rules with probabilities, so that a rule like 'friends usually like the same music' can sometimes be broken, with each rule given a weight saying how strongly it holds.
maximum a posteriori estimationPicking the single most believable setting of a model after combining the evidence with what was assumed beforehand. In practice it often amounts to ordinary training plus a penalty that keeps settings modest.
maximum likelihood estimationTuning a model's settings until the real data it learns from looks as likely, and so as unsurprising, to the model as possible. Most AI training, including the pretraining of chatbots, is a version of this.
mean average precisionThe standard score for object detectors: it rewards finding all the objects while raising few false alarms, averaged over every type of object. Higher is better, up to 100.
memorizationWhen an AI model stores specific pieces of its training data — a paragraph, a phone number, an image — closely enough to reproduce them, instead of only learning general patterns.
mesa-optimizationWhen training an AI produces a model that is itself a goal-seeking planner, with goals of its own that may differ from the ones it was trained for.
metric learningTeaching a model to measure similarity: it learns to place items so that ones that should count as alike end up close together and different ones far apart.
Donald MichieA British codebreaker turned AI pioneer who built a game-playing learning machine out of matchboxes and founded Edinburgh's influential machine-intelligence research group.
minimum description lengthThe principle that the best explanation of some data is the one that lets you write the data down most briefly, counting both the rule and its exceptions. Good theories compress.
Tom MitchellAn American computer scientist who wrote a classic machine-learning textbook with the field's most quoted definition of learning, and founded the first university department devoted to machine learning.
MLflowA free software platform that records machine-learning experiments, packages the resulting models and keeps track of which versions are ready to be used.
MLOpsThe working habits and tools that keep machine-learning models dependable inside real products once the experiment is over: automated testing, release, monitoring and retraining; the factory and maintenance crew around a model, not the model itself.
modelA simplified working version of something. In AI it usually means the trained program, a big set of learned numbers, that takes an input and produces an answer, prediction or piece of content.
model cascadeA chain of AI models ordered from cheapest to most capable: each request goes to the cheap one first and moves up the chain only if the answer does not look good enough.
model compressionThe family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy.
model monitoringContinuously checking an AI model after it goes live — what data it receives, what it predicts and whether it is still right — so that problems are caught before they cause harm.
model registryA central catalog where a team stores every version of its trained models, with records of how each was made and which version is approved for use.
model-agnosticDescribes a method that works with any kind of AI model, because it only needs to feed the model inputs and observe its outputs, never to look inside.
Monte Carlo methodEstimating an answer by running many random trials and averaging the results — like estimating the odds in a card game by dealing thousands of hands instead of calculating.
multilayer perceptronA neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model.
mutual informationHow much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated.
named-entity recognitionSoftware that spots names and similar expressions in text and labels each by type: in 'Maria flew from Lagos to Paris on Monday', it marks a person, two cities and a date.
Netflix PrizeA 2006–2009 contest in which Netflix offered a million dollars to anyone who could predict customers' movie ratings 10 percent better than its own system. It popularized machine-learning competitions and model blending.
neural architecture searchUsing an automated search, rather than human trial and error, to design the structure of a neural network: which layers to use, how many, and how to connect them.
neural tangent kernelA mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior.
NeurIPSThe largest yearly scientific conference on artificial intelligence and machine learning, held each December, where many of the field's landmark papers were first presented.
Andrew NgA British-born American computer scientist whose online courses have taught machine learning to millions; he also co-founded Google's deep learning team, Google Brain, and the online-course company Coursera.
no free lunch theoremA mathematical result saying that no learning or search method is best for every possible problem: any method that wins on some problems must lose on others.
normA way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes.
normal distributionThe familiar bell-shaped curve: most values cluster near the average, and values become rapidly rarer the farther they lie from it. Measurement errors and adult heights roughly follow it.
NumPyThe basic free Python library for working quickly with large grids of numbers; nearly every data-science and machine-learning tool in Python is built on it or copies its style.
Occam's razorA rule of thumb for choosing between explanations: if two fit the evidence equally well, bet on the one that assumes less. In AI it motivates preferring simpler models, which are less likely to fit noise.
out-of-distributionDescribes inputs that differ from anything an AI system saw in training — like a model trained on daytime photos facing a night scene — where its answers become less reliable.
overparameterizationGiving a model far more adjustable settings than it would seem to need — often more than it has training examples. Classic theory predicted this would backfire; in deep learning it usually helps.
PAC learningA mathematical definition of what it means to learn from examples: given enough data, a learner should, with high probability, find a rule that is almost always right.
pandasA free Python library for loading, cleaning, combining and summarizing tables of data, like a programmable spreadsheet; it is a standard first tool for preparing data for machine learning.
parameterOne of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned.
partial dependence plotA chart showing how a model's average prediction changes as one input is varied while everything else stays as it is in the data — for example, predicted house price against floor area.
part-of-speech taggingLabeling each word in a sentence with its grammatical role — noun, verb, adjective and so on — as in 'Time (noun) flies (verb) like (preposition) an arrow'.
PATEA privacy method in which several 'teacher' models, each trained on a separate slice of sensitive data, vote noisily to label public examples, and a 'student' model learns only from those noisy votes.
pattern recognitionThe ability of a computer, or a brain, to spot regularities in data and sort things into categories, such as recognizing handwritten letters, faces or spoken words.
Judea PearlAn Israeli-American computer scientist who gave AI the mathematics for reasoning under uncertainty and, later, for telling cause and effect apart from mere correlation.
perceptronOne of the first learning machines: a single artificial neuron that sorts inputs into two groups by drawing a straight dividing line, nudging its weights whenever it guesses wrong.
post hoc explanationAn explanation produced after a model has been built, by a separate tool that examines the finished model, rather than one that comes from a model designed to be understandable.
posteriorWhat you believe after weighing new evidence against what you believed before. It is the updated belief that Bayes' theorem produces.
predictionA model's output for a given input: its best guess at an unknown answer. In AI it need not be about the future; labeling a photo 'cat' counts as a prediction.
predictive AIAI used to forecast outcomes about people or events, such as who will repay a loan, which patient may get sicker or which machine will fail, in order to guide decisions.
predictive parityA fairness test that asks whether a system's verdict means the same thing for every group: of the people it flags as high risk, the same share really are, whatever their group.
predictive policingUsing computer programs to forecast where crimes are likely to happen, or who is likely to be involved, so that police can decide where to patrol or whom to watch.
priorWhat you believe, and how strongly, before looking at the evidence in front of you. In AI it also means built-in assumptions that steer what a system learns.
privacy-preserving machine learningThe family of techniques for training and using AI models while keeping the underlying personal or confidential data hidden — from adding careful noise to computing on encrypted data.
probabilistic graphical modelA way of describing how many uncertain quantities depend on one another by drawing them as a network, which makes it practical for a computer to reason about all of them together.
probabilistic programmingWriting a statistical model as an ordinary-looking program that includes random choices, and letting the system work out automatically which hidden values best explain the observed data.
probabilityHow likely something is, on a scale from 0 for the impossible to 1 for the certain; rolling a six with a fair die scores one-sixth. AI systems state their predictions, even each next word, as probabilities.
probability density functionA curve showing where a measurement is likely to fall. The area under the curve between two values gives the chance of landing between them.
probability distributionA complete description of how likely each possible outcome is — for a fair die, one-sixth for each face. A language model produces one of these over its whole vocabulary at every step.
probingTesting what information is hidden inside an AI model by training a small, separate predictor on the model's internal activity: if the predictor can recover something, the model has stored it somewhere.
protected attributeA personal trait, such as race, sex, religion, age or disability, that the law says must not be used to treat people unfairly.
proxy discriminationDiscrimination that happens when a system cannot see a trait like race directly but relies on something closely linked to it, such as a postcode, and so treats groups unequally anyway.
pruningCutting away the parts of an AI model that contribute least, as a gardener trims branches, so the model becomes smaller or faster while still working nearly as well.
quantum machine learningResearch into whether quantum computers could learn from data faster or better than ordinary computers, and into using AI to study quantum systems. Practical advantages have not yet been shown.
Ross QuinlanAn Australian computer scientist who created ID3 and C4.5, influential programs that learn decision trees, flowchart-like rules for classifying examples.
radial basis function networkA three-layer network whose hidden neurons each respond to inputs near a particular reference point, like bells placed across the data; the output is a weighted mix of the bells.
random variableA quantity whose value depends on chance, such as the number rolled on a die or the next word a chatbot will write.
regretHow much worse a learner did than it could have done had it known the best choice from the start — the price of having to learn by trying.
representation learningLetting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
rerankingA second, more careful pass over search results: a fast method grabs a few hundred candidates, then a slower, smarter model reorders them so the best come first.
robot learningTeaching robots skills from data and experience, such as watching demonstrations, practicing by trial and error or training in simulation, instead of programming every movement by hand.
robustnessAn AI system's ability to keep working correctly and safely when conditions change — unfamiliar data, noisy inputs, or people deliberately trying to trick it.
sample efficiencyHow much practice an AI needs to learn something — an agent that learns a game in a hundred tries is far more sample-efficient than one that needs millions.
Arthur SamuelAn American IBM engineer whose checkers program improved by playing against itself, and whose 1959 paper helped make 'machine learning' the name of a field.
Samuel's checkers programA 1950s IBM program by Arthur Samuel that learned to play checkers better than its author, partly by playing against itself. It was among the first programs to learn, and its 1959 paper popularized the phrase 'machine learning'.
scikit-learnA free, widely taught Python library of classic machine-learning methods — such as decision trees, regression and clustering — that all work in the same simple way.
secure aggregationA cryptographic trick that lets a server add up updates from many devices while learning only the total, never any single device's contribution.
self-organizing mapA neural network that arranges data on a flat grid so that similar items end up in neighboring cells, producing a map of how the data is organized.
self-supervised learningA way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image.
semi-supervised learningMachine learning that combines a small set of labeled examples with a much larger pile of unlabeled ones, which are cheaper to collect, to learn better than from the labeled data alone.
sentiment analysisUsing software to judge the feelings or opinions expressed in text — whether a review, post or comment is positive, negative or neutral, and about what.
SHAPA popular method that explains a single prediction by fairly dividing it among the inputs, using a rule from game theory for splitting a team's winnings among its players.
Shapley valueA rule from game theory for splitting a shared reward fairly among team members, by averaging how much each member adds when joining the team in every possible order.
shortcut learningWhen an AI system gets right answers for the wrong reasons — noticing which hospital an X-ray came from rather than signs of disease — so it fails once that accidental clue disappears.
Siamese networkA matching machine built from twin copies of one neural network: each copy turns one of two inputs, say two signatures or two face photos, into a summary, and the two summaries are compared.
sigmoid functionAn S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on.
simulated annealingAn optimization method inspired by slowly cooling metal: early on it sometimes accepts worse solutions, to escape dead ends, and becomes pickier as a 'temperature' setting gradually drops.
singular value decompositionA way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data.
sparse retrievalFinding documents by the words they contain, giving more weight to rare, telling words — the classic way search engines work.
spurious correlationA link in data that fools you: two things move together, yet neither drives the other, because a hidden shared cause, the way the data were collected or luck produced the pattern. AI models readily latch onto such links.
statistical learning theoryThe mathematics of when and why learning from examples works: how much data is needed, and how well a model trained on some examples will do on new ones.
stochasticInvolving chance, so the same starting point can lead to different outcomes. AI chatbots are stochastic: ask the same question twice and you may get two different answers.
stochastic gradient descentA faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset.
structural causal modelA precise description of what causes what in a system — a diagram of arrows plus equations — that predicts what would happen if you intervened, not just what tends to occur together.
supervised learningTeaching a computer by example: you give it many inputs paired with the right answers, such as photos labeled 'cat' or 'dog', and it learns to answer correctly for new inputs.
surrogate modelA simpler stand-in model trained to imitate a more complex one — to explain it, or to approximate a slow or costly calculation cheaply.
symbolic regressionGetting a computer to discover a readable mathematical formula, such as a law of physics, that fits a set of measurements, instead of only fitting numbers into a formula chosen in advance.
task (machine learning)The job an AI model is meant to do, such as translating sentences, spotting spam or answering questions, defined by its inputs, its expected outputs and how success is scored.
text classificationSorting pieces of text into categories automatically — what topic an article covers, whether a message is spam, which team should handle a customer's complaint.
TF-IDFA simple score for picking out the words that characterize a document: a word counts for more the more often it appears there, and for less the more of the collection's other documents also use it.
trainingThe process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better.
training–serving skewA mismatch between how data is prepared when a model is trained and when it is used for real, which makes the live model quietly worse than its tests promised.
transfer learningReusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data.
uncertainty quantificationMeasuring how unsure an AI system's predictions are, and why, so that people know when to trust an answer and when to double-check it.
unsupervised learningMachine learning without answer keys: the computer looks at unlabeled data and finds structure on its own, such as groups of similar customers or the main ways in which data vary.
Leslie ValiantA British computer scientist who in 1984 gave the first rigorous definition of what it means for a computer to learn from examples, founding the theory of machine learning.
Vladimir VapnikA Soviet-born American mathematician who built the statistical theory of how machines learn from examples and co-invented support vector machines, a leading method before deep learning.
varianceA number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves.
variational inferenceA way of approximating a hard-to-compute probability distribution by picking the closest match from a simpler family of distributions, turning a hard problem of adding up every possibility into an optimization problem.
VC dimensionA measure of how flexible a family of prediction rules is: the largest number of points it can label in every possible way. More flexible families need more data to learn reliably.
Wasserstein distanceA way of comparing two spreads of probability by imagining one as heaps of sand that must be shoveled into the shape of the other: the less shoveling, weighted by distance, the more alike they are.
weak supervisionTraining AI with labels that are cheap but imperfect, such as rough rules of thumb, partial tags or noisy guesses, instead of carefully hand-checked answers.
weak-to-strong generalizationThe finding that a strong AI trained only on answers from a weaker, error-prone teacher can end up outperforming that teacher — studied as a stand-in for humans supervising smarter-than-human AI.
weight decayA training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data.
Weights & BiasesA company and online service that machine-learning teams use to log, chart and compare their training experiments, models and AI-app test results in shared dashboards.
XOR problemThe classic puzzle of teaching a single artificial neuron the 'one or the other but not both' rule. It cannot be done with one straight dividing line, which showed why networks need hidden layers.
zero-shot learningDoing a task without being shown a single example of it — say, an AI sorting reviews into positive and negative just from being told to, or recognizing an animal it never saw in training.