Deep Learning

Neural networks: their layers and architectures, and how they are trained. 156 entries live here, and 354 from other domains touch it.

Essential

artificial neuronThe basic building block of a neural network: a tiny calculator that multiplies each incoming number by a weight, adds them up, and passes the total through a simple function to decide its output.
backpropagationThe method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
embeddingA list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance.
gradient descentA method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.
layerOne stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage.
loss functionA score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down.
transfer learningReusing what a model learned on one task to get a head start on another — like a pianist picking up the organ faster than a beginner. It lets models learn new jobs from little data.

All other entries

149

activation functionThe simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer.
AdafactorA memory-saving cousin of the popular Adam training method. Instead of keeping extra statistics for every single weight, it keeps summaries for each row and column of a weight table.
AdaGradA training method that gives each of a model's settings its own step size, shrinking steps for settings that have already had many large updates and keeping them larger for rarely updated ones.
AdamThe most widely used method for training neural networks. It smooths the direction of updates over time and automatically sets a separate step size for each setting based on how large its recent changes have been.
AdamWA corrected version of the popular Adam training method that applies its steady pull toward smaller weights separately from the main update. It is the standard choice for training large language models.
associative memoryA memory that is searched by content rather than by location: given part of a stored pattern, or a noisy copy of it, it recalls the whole, much as a few notes can bring back an entire song.
autoencoderA neural network trained to squeeze its input into a compact code and then rebuild the original from that code. Learning to compress this way forces it to capture what matters in the data.
automatic differentiationA technique that lets a computer work out exactly how a program's output would change if its inputs changed slightly, by tracking every arithmetic step. It lets AI software compute training updates automatically.
backpropagation through timeThe way recurrent networks are trained: the network is unrolled into a long chain, one copy per time step, and errors are traced back through every step to adjust the shared weights.
backward passThe step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change.
batch normalizationA layer that rescales the numbers flowing through a network so that, across each batch of training examples, they keep a steady average and spread. It makes deep networks train faster and more reliably.
batch sizeHow many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step.
Bayesian neural networkA neural network that keeps a whole range of plausible settings for its internal numbers instead of one best guess, so that it can say how unsure it is about each prediction.
benign overfittingWhen a model fits its training data perfectly — even the errors and noise — yet still predicts well on new data, defying the textbook warning that this always backfires.
bidirectional recurrent neural networkA recurrent network that reads a sequence both forwards and backwards, so its understanding of each word or sound can draw on what comes after it as well as what came before.
capsule networkA network design championed by Geoffrey Hinton in which groups of neurons, called capsules, each describe a part of an object and its position, so the network can check whether the parts fit together.
computational graphA map of a calculation drawn as boxes and arrows: each box is a simple operation and each arrow carries a number to the next step. AI software uses it to trace errors backward.
connectionist temporal classificationA training method that lets a network learn to transcribe speech or handwriting without being told exactly where each letter or sound begins and ends.
contrastive learningA way to learn without labels by comparison: the model learns to pull together two versions of the same thing, such as two crops of one photo, and push apart unrelated things.
convergenceThe point at which training settles down and further steps no longer meaningfully improve the model — and, in theory, a guarantee that a method will reach a solution, and how fast.
ConvNeXtA 2022 image-recognition network showing that an older-style convolutional design, updated with lessons from transformers, can match vision transformers.
convolutionAn operation that slides a small grid of weights across an image or signal, multiplying and adding at each spot, to measure how strongly a pattern appears at every location.
convolutional neural networkA kind of neural network built for images and other grid-like data. It slides small pattern detectors across the input, finding edges first, then shapes, then whole objects.
cosine annealingA popular plan for shrinking a model's step size during training: steps start large, shrink slowly, then faster, then level off near zero, following the shape of half a wave.
curriculum learningTeaching a model the way schools teach people: starting with easy examples and gradually moving to harder ones, instead of presenting everything in random order.
data augmentationCreating extra training examples by changing existing ones in ways that keep their meaning — a flipped, cropped or recolored photo of a cat still shows a cat — so the model learns more robustly.
denoising autoencoderAn autoencoder trained to recover clean data from a deliberately damaged copy, such as an image with random pixels erased, which teaches it the structure of the data rather than how to copy.
DenseNetAn image-recognition network in which every layer receives the outputs of all the layers before it, so features are reused rather than relearned.
depthwise separable convolutionA cheaper way to do the main operation of image networks: first scan each channel of the input separately, then mix the channels together with a tiny one-by-one step.
dilated convolutionA version of the scanning-window operation that spreads its weights apart with gaps, so it can take in a wider area of an image or sound at no extra cost.
disentangled representationAn internal representation in which each part tracks one separate, meaningful factor of the data, such as one number for a face's angle, another for its lighting and another for hair color.
double descentA surprising pattern where a model's error on new data first falls, then rises, then falls again as the model grows bigger — contradicting the old rule that oversized models always do worse.
dropoutA training trick that randomly switches off a fraction of a network's neurons at each step, so the network cannot lean too heavily on any one of them and learns sturdier patterns.
dying ReLUA training failure in which some of a network's units get stuck permanently switched off, always outputting zero, so they stop learning and become dead weight.
early stoppingStopping training at the point where the model does best on practice data it has not learned from, before it starts memorizing quirks of its training examples.
edge of stabilityA surprising pattern in how neural networks train: the error landscape keeps getting steeper until it is just steep enough to make training wobble, then stays balanced at that edge while the error still falls.
EfficientNetA family of image-recognition networks from 2019 that grows a small base design in a balanced way, deeper, wider and sharper together, to get more accuracy for each unit of computing.
encoder–decoder architectureA two-part model design: one part, the encoder, reads the input and condenses it into an internal summary; the other, the decoder, uses that summary to produce the output.
energy-based modelA model that scores how well things fit together by giving each possibility an 'energy': low for plausible combinations, high for implausible ones, like a ball settling into the lowest valleys.
epochOne complete pass through all the training examples. Smaller models are often trained for many epochs, seeing each example many times; the largest language models may see most of their data only once.
equivarianceA property of a model whose output changes in step with its input: move the cat in a photo, and the network's internal 'cat' signal moves the same way.
exploding gradient problemA training failure in which the learning signal grows out of control as it passes back through a network, so the weights jump wildly and training breaks down.
exponential moving average of weightsA smoothed copy of a model kept during training, which each update nudges only slightly toward the latest weights. The calmer copy often performs better than the jittery original.
exposure biasA weakness of models trained only on perfect examples: when they write on their own, one early mistake puts them in unfamiliar territory they never practiced, and errors can snowball.
feature mapThe grid of numbers produced when one learned pattern detector is scanned across an image, showing where, and how strongly, that pattern appears.
feedforward neural networkA neural network laid out like an assembly line: each stage hands its result only to later stages, never back, so every answer depends on the current input alone and nothing is remembered between inputs.
filterIn an image-recognition network, a small grid of learned numbers that acts as a pattern detector, for vertical edges, a patch of fur or a wheel, scanned across the whole picture.
focal lossA way of scoring a model's mistakes during training that pays less attention to examples it already gets right easily, so that learning concentrates on the hard and rare cases.
forward passThe step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction.
fully connected layerA layer in which every input is connected to every output by its own adjustable weight. It is the most general and most common kind of neural-network layer.
fully convolutional networkAn image network made only of scanning-window layers, with no fixed-size final layers, so it can take pictures of any size and label every pixel.
gated linear unitA layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through.
gated recurrent unitA simpler cousin of the LSTM: a recurrent network unit with two learned gates that decide how much of its memory to keep and how much to overwrite at each step.
GELUA smooth version of the ReLU rule, used in models such as BERT and GPT: it keeps large positive values, removes large negative ones, and blends softly in between.
generalization gapThe difference between how well a model does on the examples it trained on and how well it does on new ones. A big gap means it memorized rather than learned.
geometric deep learningAn approach to designing neural networks around the shapes and symmetries of the data, whether grids, graphs, surfaces or 3-D structures, so the network respects how that data can be moved or rearranged.
gradient accumulationA way to train with large batches on limited hardware: the computer processes several small batches one after another, adds up what each says, and only then makes a single update.
gradient checkpointingA memory-saving trick: instead of storing every intermediate result needed for learning, the computer keeps only a few and recalculates the rest when needed, trading extra computing time for memory.
gradient clippingA safety limit on how big a single training update can be. If the computed adjustment is unusually large, it is scaled down so that one bad batch cannot wreck the model.
graph neural networkA neural network for data shaped like a web of connections, such as molecules, social networks or road maps, in which each point repeatedly gathers information from its neighbors.
grokkingWhen a model that has merely memorized its training examples suddenly, long afterward, figures out the underlying rule and starts getting new examples right.
Gumbel-softmaxA mathematical trick that lets a neural network make a choice among separate options, such as picking one word or one category, while still learning through small, smooth adjustments.
hidden layerAny layer of a neural network between the input and the output. Its values are never shown to the user; they are the network's internal working-out.
hierarchical reasoning modelA small, brain-inspired AI model that solves puzzles with two linked parts — a slow planner and a fast worker — instead of writing out its reasoning in words.
highway networkAn early design for very deep networks in which learned gates let information either pass through a layer unchanged or be transformed, making networks hundreds of layers deep trainable.
hypernetworkA neural network whose job is to produce the settings, or weights, of another neural network, so that one model can generate many specialized models on demand.
implicit neural representationA way of storing an image, shape or scene as a small neural network that, given any coordinate, returns what is there, such as a color or density, instead of as a grid of pixels.
implicit regularizationThe hidden preferences of a training method itself: even with no explicit rule against complexity, the way a model is trained tends to steer it toward simpler solutions.
InceptionA family of Google image-recognition networks, the first called GoogLeNet, whose building blocks examine each patch of an image at several scales at once while keeping computation low.
invarianceA property of a model whose answer does not change when the input is altered in an irrelevant way, such as a photo classifier that still says 'cat' when the cat moves or the picture is flipped.
Kolmogorov–Arnold networkA 2024 alternative to the standard neural network: instead of fixed rules inside neurons, every connection learns its own small curve, which can make the result easier to read as a formula.
label smoothingA training trick that tells a model the correct answer is very likely rather than absolutely certain, which stops it from becoming overconfident and often helps it generalize.
latent spaceThe hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates.
layer normalizationA layer that rescales the numbers inside a network, separately for each example, so they keep a steady average and spread. It, or a simpler variant, is a standard part of the models behind today's chatbots.
leaky ReLUA variant of the common ReLU rule that lets a small fraction of each negative value through instead of zeroing it, so neurons never go completely silent.
learning curveA graph showing how a model's performance improves as it gets more training — more examples or more time. Its shape tells you whether more data or a bigger model would help.
learning rateHow big a step a model takes each time it adjusts its settings during training. Too large and learning goes haywire; too small and it crawls.
learning rate scheduleA plan for changing how big a model's learning steps are during training — usually starting cautiously, then taking big steps, then smaller and smaller ones as it settles in.
learning rate warmupStarting training with tiny learning steps and gradually increasing them over the first stretch of training, so the model does not lurch wildly before it has found its footing.
LionA training method discovered by a computer search over possible algorithms. It moves every setting by the same small amount at each step, choosing only the direction, and needs less memory than the popular Adam method.
liquid neural networkA small, brain-inspired network for time-varying data whose neurons change how quickly they react depending on their input, letting a few neurons handle tasks such as steering a car.
logitA raw score that a model gives each possible answer before the scores are turned into probabilities. In statistics, the 'log-odds': a probability rewritten on a scale running from minus to plus infinity.
long short-term memoryA type of recurrent network with a separate memory lane and learned 'gates' that decide what to remember, forget and output, letting it keep track of information over long sequences.
loss landscapeA way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys.
lottery ticket hypothesisThe idea that a large, randomly started neural network contains a much smaller network inside it that, trained alone from the same starting values, would learn just as well — a winning lottery ticket.
metric learningTeaching a model to measure similarity: it learns to place items so that ones that should count as alike end up close together and different ones far apart.
mini-batchA small random handful of training examples that a model learns from in a single step. Training works through many such handfuls instead of studying the whole dataset at once.
mixed-precision trainingTraining a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster.
MobileNetA family of compact image-recognition networks from Google, designed to run quickly on phones and other small devices with little loss of accuracy.
mode connectivityThe discovery that two separately trained versions of a network, though their settings look very different, can usually be joined by a path along which the model keeps working well.
momentumA training trick that lets each update keep part of the previous update's direction, like a ball rolling downhill that builds up speed and coasts over small bumps.
multilayer perceptronA neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model.
MuonA newer training method for neural networks that tidies each update so it pushes evenly in all directions within a weight table. Several groups report that it trains language models with less computing than the standard method.
nested learningA Google research idea that treats an AI model as layers of learning processes updating at different speeds, with the aim of letting models keep learning without forgetting old skills.
neural architecture searchUsing an automated search, rather than human trial and error, to design the structure of a neural network: which layers to use, how many, and how to connect them.
neural collapseA striking tidiness that appears when classifiers are trained for a long time: the network's internal summaries of each class shrink to single points arranged as evenly and as far apart as possible.
neural operatorA neural network that learns how a whole physical system responds, such as how a fluid flows given its starting state, so it can stand in for slow simulations and answer new cases quickly.
neural ordinary differential equationA neural network that, instead of passing data through a fixed stack of layers, describes how the data changes continuously over 'time' and uses an equation solver to follow that change.
neural tangent kernelA mathematical tool showing that extremely wide neural networks learn like a simpler, well-understood method, because their internal settings barely move during training. It explains some, but not all, of deep learning's behavior.
neural Turing machineA neural network attached to an external memory bank that it learns to read from and write to, an early attempt to give networks something like a computer's working memory.
normalization layerA layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training.
optimizerThe rule a training program uses to turn measured errors into actual changes to a model's internal numbers, deciding the direction and size of each adjustment.
overparameterizationGiving a model far more adjustable settings than it would seem to need — often more than it has training examples. Classic theory predicted this would backfire; in deep learning it usually helps.
perceptronOne of the first learning machines: a single artificial neuron that sorts inputs into two groups by drawing a straight dividing line, nudging its weights whenever it guesses wrong.
physics-informed neural networkA neural network that is penalized during training whenever its output breaks a known physical law, such as an equation of fluid flow, so it can model a system well even from few measurements.
poolingA step in an image network that shrinks its internal picture by summarizing each small patch with one number, usually the largest or the average, keeping what matters and discarding exact positions.
radial basis function networkA three-layer network whose hidden neurons each respond to inputs near a particular reference point, like bells placed across the data; the output is a weighted mix of the bells.
receptive fieldThe patch of the original input, such as a region of an image, that a particular neuron inside a network can see and respond to.
recurrent neural networkA neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood.
ReLUThe most common rule a neuron uses to produce its output: keep the number if it is positive, replace it with zero if it is negative. Simple, fast and surprisingly effective.
representation learningLetting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
reservoir computingA shortcut way to build a network for time-varying signals: a large, fixed, randomly wired network echoes the input, and only a simple final readout is trained.
residual connectionA shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable.
ResNetA family of image-recognition networks from 2015 whose shortcut connections allowed far deeper networks than before, over 100 layers, and became a standard building block across AI.
restricted Boltzmann machineA two-layer network, one layer for the data and one hidden, that learns the patterns in data by trying to generate it. Stacks of them helped start the deep-learning revival around 2006.
RMSNormA simpler, faster version of the layer that keeps a network's internal numbers at a steady scale. Most recent large language models use it.
RMSPropA training method that scales each setting's step by how large its recent changes have been, so jumpy settings move cautiously and quiet ones move faster.
schedule-free optimizationA way of training that needs no plan for slowing learning down over time, so training can be stopped or extended at any point and still produce a well-settled model.
second-order optimizationTraining methods that use not only the slope of the error but also how that slope is changing — its curvature — to choose better-sized steps. They can need far fewer steps, but each costs more.
self-organizing mapA neural network that arranges data on a flat grid so that similar items end up in neighboring cells, producing a map of how the data is organized.
sequence-to-sequence modelA model that turns one sequence into another, such as a French sentence into English or speech into text, by first reading the whole input and then writing the output step by step.
ShampooA training method that adjusts each weight table's updates using statistics gathered along its rows and its columns, capturing how weights interact. Each step costs more computing, but fewer steps are needed.
sharp and flat minimaTwo kinds of low points a trained model can settle into: narrow pits where small changes spoil performance, and broad basins where they barely matter. Broad basins are thought to generalize better, though this is debated.
sharpness-aware minimizationA training method that looks for settings where the error stays low even if the settings are nudged a little, rather than settings that are merely low at one precise point.
Siamese networkA matching machine built from twin copies of one neural network: each copy turns one of two inputs, say two signatures or two face photos, into a summary, and the two summaries are compared.
sigmoid functionAn S-shaped curve that squeezes any number into a value between 0 and 1, so the result can be read as a probability or as how far a switch is turned on.
SiLUA smooth activation rule that multiplies each value by a soft on/off switch computed from the value itself. It is close to ReLU but curved, and is used in many modern models.
softmaxA formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.
spiking neural networkA neural network that, like the brain, communicates with brief pulses, or spikes, sent only when a neuron's charge crosses a threshold. On special chips this can save a great deal of energy.
squeeze-and-excitation networkAn image-network design whose add-on blocks let the network decide, for each picture, which of its internal feature channels to turn up and which to turn down.
stochastic gradient descentA faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset.
straight-through estimatorA shortcut that lets networks learn even when they contain steps with no useful slope, such as rounding: during learning, the computer simply pretends the step passed its input straight through.
strideHow far a scanning window jumps each time it moves across an image. Jumping two pixels instead of one halves the size of the result.
SwiGLUA design for the feed-forward layers inside most modern language models: one calculation produces values, another produces a smooth on/off switch, and the two are multiplied together.
tanhAn S-shaped curve, like the sigmoid but centered on zero, that squeezes any number into a value between −1 and 1. It was the standard neuron rule before ReLU took over.
teacher forcingTraining a model that writes one word at a time by always feeding it the correct previous words, rather than its own possibly mistaken guesses, while it learns to predict the next one.
time delay neural networkAn early network for speech that applied the same small pattern detector at every moment of a sound, so it could recognize a sound wherever it occurred in time.
tiny recursive modelA very small AI model that solves hard puzzles by going over its own answer again and again, improving it each time, instead of relying on sheer size.
training loopThe repeating cycle at the heart of training software: grab some examples, let the model guess, measure the error, adjust the model, and do it again — often millions of times.
training stepOne round of learning: the model makes predictions on a small batch of examples, measures its error and adjusts its settings once. Training runs consist of thousands to millions of steps.
transposed convolutionAn operation that roughly reverses an image network's shrinking steps, enlarging a small grid of features into a bigger one, as when a model paints a full-size image.
U-NetAn image network shaped like the letter U: one half shrinks the picture to understand it, the other enlarges it back, with shortcuts between the halves so fine detail is not lost.
universal approximation theoremA mathematical result showing that a neural network with enough neurons can imitate essentially any smooth input-to-output relationship as closely as you like, though it does not say how to find that network.
vanishing gradient problemA training difficulty in deep or recurrent networks: the learning signal shrinks as it travels back through many layers or time steps, so the early parts of the network barely learn.
VGGNetAn influential 2014 image-recognition network from Oxford, built by stacking many identical small pattern detectors, which showed that making networks deeper by itself improves accuracy.
weight decayA training rule that shrinks every weight slightly at each step, like a small tax on large numbers. It keeps models simpler and usually helps them work better on new data.
weight initializationChoosing the random starting values of a neural network's internal numbers before training. Good starting values keep signals from fading away or blowing up as they pass through many layers.
weight sharingReusing the same learned numbers in many places in a network, for example one edge detector applied at every spot in an image, so the network needs fewer parameters and learns from less data.
XOR problemThe classic puzzle of teaching a single artificial neuron the 'one or the other but not both' rule. It cannot be done with one straight dividing line, which showed why networks need hidden layers.

Also relevant

354

A3CA 2016 DeepMind training method in which many copies of an AI agent learn in parallel on separate copies of a task, pooling their updates into one shared neural network.
action recognitionTeaching computers to recognize what people are doing in a video, such as running, waving, cooking or falling, rather than just which objects appear in it.
activationThe numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment.
activation spaceThe imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions.
activation vectorThe list of numbers a neural network produces at one layer for one input — a snapshot of its internal state. In steering, also a direction added to that state to shift behavior.
ADALINEAn early learning machine from 1960 that adjusted its settings to shrink its errors step by step, the same basic idea used to train today's neural networks.
adapterA small trainable add-on slotted into a large frozen AI model to teach it a new task or style, like a plug adapter that lets one device work in many countries.
adversarial exampleAn input altered on purpose, often too slightly for people to notice, so that an AI model gets it badly wrong — like a panda photo that a classifier confidently calls a gibbon.
adversarial machine learningThe study of how machine-learning systems can be fooled, corrupted or spied on by someone working deliberately against them, and of how to build systems that hold up.
adversarial robustnessHow well an AI model keeps giving correct answers when someone deliberately tweaks its inputs to trip it up.
adversarial trainingToughening an AI model by deliberately showing it attacks during training — the tricky inputs that fool it — along with the right responses, so it learns to resist them.
AI acceleratorAny chip built or tuned to speed up AI — graphics chips, Google's TPUs, phone 'neural engines' and many startup designs. Such chips do the heavy math of training and running models.
AlexNetThe neural network that won a major image-recognition contest by a huge margin in 2012, convincing researchers and companies that deep learning trained on graphics chips was the way forward.
AlgoPerfA public contest that measures which methods for training neural networks reach a set level of accuracy fastest on identical hardware, across several kinds of tasks, without being hand-tuned for each one.
ALiBiA simple way to give a transformer a sense of word order: attention between two words is penalized more the farther apart they are, which helps it handle texts longer than those it trained on.
AlphaFoldA DeepMind AI system that predicts the 3D shape of proteins from their chemical sequence, often about as accurately as laboratory experiments. It cracked a problem biologists had worked on for 50 years and won a Nobel Prize.
AlphaGoA DeepMind program that in 2016 beat Lee Sedol, one of the world's best players of the board game Go, a decade or more before many experts expected a computer to manage it.
AlphaGo ZeroA program from DeepMind that taught itself to play the board game Go better than any human in 2017, starting from nothing but the rules and learning only by playing millions of games against itself.
Shun-ichi AmariA Japanese mathematician who trained multilayer neural networks by gradient descent as early as 1967 and founded information geometry, the source of the 'natural gradient' method.
anchor boxA ready-made box shape, placed at many spots across an image, that an object detector stretches and shifts to fit real objects instead of drawing boxes from scratch.
annealingThe last stretch of training, when the model's learning is slowed to a stop — often while it is fed a small helping of the best data — so it settles into its final, sharpest version.
architectureThe overall design of an AI model, meaning what kinds of parts it has and how they are connected, as opposed to the specific numbers it learns during training.
attentionA way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones.
attention headOne of several parallel attention units inside a transformer layer. Each head learns its own way of deciding which words to connect, so different heads can specialize in different patterns.
Attention Is All You NeedThe 2017 Google research paper that introduced the transformer, the neural-network design behind ChatGPT, Claude, Gemini and nearly every other modern AI language system.
attention maskA grid of yes-or-no switches telling an attention layer which words each word may look at — for example, ignore blank padding, or do not peek at words that come later.
attention-based explanationExplaining an AI model's output by showing which words or image patches its attention mechanism weighted most — an intuitive but disputed guide to what actually drove the output.
automated circuit discoverySoftware methods that search a neural network automatically for the small set of internal connections responsible for a chosen behavior, instead of a researcher hunting for them by hand.
autoregressive image generationMaking a picture one piece at a time, each piece predicted from those already made — the way chatbots write text word by word.
backboneThe main, usually pretrained part of a vision model that turns an image into useful internal features; smaller task-specific parts are attached to it, like limbs to a spine.
backdoor attackSecretly training an AI model to misbehave whenever it sees a hidden signal — a sticker, a word, a date — while it acts perfectly normally the rest of the time.
Dzmitry BahdanauA Belarusian computer scientist who, as a student in Yoshua Bengio's lab in 2014, was lead author of the paper that introduced attention, the mechanism later at the core of the transformer.
Yoshua BengioA Canadian computer scientist who helped found deep learning, including ideas behind today's language models, and who has become a leading voice on the risks of advanced AI.
bfloat16A 16-bit number format invented at Google for AI. It keeps the huge range of the standard 32-bit format but drops precision, which neural networks can usually tolerate.
bias–variance tradeoffThe balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.
bi-encoderAn AI design that turns a question and a document into number lists separately, so documents can be processed once in advance and compared with any new question in a flash.
binding problemThe puzzle of how a brain, or a neural network, keeps track of which features belong together — that the red goes with the apple and the green with the leaf — when they are processed separately.
black-box attackAn attack on an AI model by someone who cannot see inside it and can only send it inputs and watch what comes out — like picking a lock by feel.
Boltzmann machineA network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them.
byte-level modelA language model that reads raw bytes — the basic units computers use to store text — instead of tokens from a fixed list, avoiding a separate tokenizer but making inputs much longer.
CaffeAn early free deep-learning toolkit from the University of California, Berkeley, popular for image recognition in the mid-2010s, in which networks were described in simple text files rather than code.
Carlini–Wagner attackA careful, optimization-based way of finding the smallest change to an input that fools a neural network; it broke several defenses that had looked secure.
catastrophic forgettingWhen a neural network trained on something new gets suddenly and badly worse at what it could already do, because the new training rewrites the internal settings the old skills relied on.
causal maskA rule inside text-generating AI that stops each word from seeing the words after it, so the model learns to predict what comes next without being able to cheat.
centered kernel alignmentA score from 0 to 1 measuring how similar two neural networks' internal representations are, by comparing how each arranges the same set of inputs relative to one another.
certified robustnessA mathematical guarantee — not just a test result — that no change to an input within a set limit can alter a model's answer.
chain ruleA rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer.
Kyunghyun ChoA South Korean computer scientist who co-invented the encoder–decoder approach to neural machine translation and the gated recurrent unit, and co-wrote the paper that introduced attention.
François CholletA French software engineer who created Keras, a popular tool for building neural networks, and the ARC benchmark, which tests whether AI can solve kinds of puzzles it has never seen.
circuitA small, identifiable piece of wiring inside a neural network — a few internal parts and the connections between them — that together carry out one specific job, such as spotting curves or copying a name.
classifier guidanceA way to push a diffusion image generator toward a requested category by consulting a separate image-recognition model at every step, trading variety for images that more clearly show the target.
classifier-free guidanceA setting that makes an image generator follow the prompt more strictly by comparing what it would draw with and without the prompt, then exaggerating the difference.
clean-label poisoningA sneaky form of data poisoning in which every planted training example is labeled correctly, so reviewers see nothing wrong, yet the examples are crafted to make the model fail.
CLIPAn OpenAI model that learned to match pictures with their captions from hundreds of millions of web examples, so it can judge how well any piece of text describes any image.
computational intelligenceA family of AI methods inspired by nature, such as neural networks, fuzzy logic and evolutionary algorithms, that learn or adapt from numbers rather than following explicit logical rules.
ConformerA neural-network design for speech recognition that combines a transformer, which relates distant parts of a recording, with convolutions, which pick up fine local sound patterns.
connectionismThe view that minds and intelligent machines are best built from huge networks of simple, neuron-like units whose connection strengths are learned, rather than from explicit rules and symbols.
connectomeA full chart of how a nervous system is wired: which nerve cells, or which brain regions, are linked to which others.
continual learningThe ability of an AI to keep learning new things over time from a stream of experience, without forgetting what it already knew and without being retrained from scratch.
ControlNetAn add-on for image generators that makes them follow a guide image — a sketch, a stick-figure pose or a depth map — so the layout comes out as intended.
credit assignment problemThe puzzle of figuring out which of many earlier decisions, or which parts of a system, deserve the credit or blame for a final outcome — like working out which moves won a chess game.
critical batch sizeThe largest number of examples a model can usefully learn from in each training step. Below it, bigger batches let many chips work in parallel and finish sooner; above it, the extra chips are mostly wasted.
cross-attentionAttention between two different inputs: one sequence consults another, as when a translation model writing French looks back at the English sentence, or an image generator consults the text prompt.
crosscoderA concept-finding tool that looks at several layers, or several versions of a model, at once, so that one shared concept can be tracked wherever it appears.
cross-encoderAn AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list.
cross-entropyA score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
CUDANvidia's software system for programming its graphics chips to do general calculations. Most AI software is built on it, which is a big reason Nvidia chips dominate AI.
Jeff DeanAn American computer scientist who built much of Google's computing infrastructure, co-founded the Google Brain deep-learning team and was Google's chief scientist before leaving in 2026 to co-found a startup.
decision transformerA way of solving decision problems with the same kind of model that powers chatbots: it reads past situations, actions and the score it is asked to achieve, and predicts the next action.
decoder-only transformerThe kind of transformer behind most chatbots: a single stack that reads text left to right and continues it, without a separate part that first encodes an input.
deep belief networkA many-layered neural network trained one layer at a time, introduced in 2006. It showed that deep networks could be trained after all and helped launch the deep learning boom, before simpler methods replaced it.
deep learningMachine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract.
deep learning revolutionThe rapid takeover of AI by many-layered neural networks from about 2012, when they suddenly began beating older methods at recognizing images and speech, leading to today's AI systems.
deep Q-networkA method, made famous by DeepMind in 2013–2015, in which a neural network learned to play dozens of Atari video games from screen pixels and the score alone, many at human level.
deep reinforcement learningReinforcement learning in which a large neural network does the learning, letting an AI learn tasks such as video games or robot control directly from raw inputs like screen pixels or camera images.
DeepLIFTA method that explains a neural network's output by comparing each internal signal with its value on a neutral reference input, then passing the differences back to the inputs as credit.
delta ruleA simple learning rule for one artificial neuron: adjust each connection in proportion to the error, the gap between the wanted and the actual output, times that connection's input.
denoising diffusion probabilistic modelThe 2020 recipe that made diffusion image generators work well: add noise to pictures over a thousand small steps, then train a network to guess the noise so it can be removed.
depth estimationWorking out how far away each point in a scene is from the camera, producing a 'depth map' that turns a flat picture into a sense of near and far.
derivativeHow fast something changes as one of its inputs changes — the steepness of a curve at a point. Training an AI means repeatedly following such slopes downhill.
DETRAn object detector built on the transformer, the design behind modern chatbots, that outputs a short list of objects in one go instead of sorting through thousands of guessed boxes.
diffusion modelAn AI that learns to create images, sound or video by practicing noise removal: shown pictures buried under static, it learns to clean them up, then builds new ones starting from pure static.
diffusion transformerA diffusion image or video generator whose core network is a transformer — the architecture behind chatbots — instead of the older image-specific design, which lets it improve more predictably as it grows.
DiLoCoA training method from Google DeepMind that lets groups of chips in different places train one language model while talking to each other hundreds of times less often than usual.
dimensionalityHow many separate numbers are needed to describe something. A point on a map needs two; an AI's description of a single word may use thousands.
DINOA family of Meta vision models that learn to understand images from millions of unlabeled photos, by training one network to agree with a slowly updated copy of itself.
domain adaptationAdjusting a model trained on one kind of data, such as daytime photos or one hospital's records, so that it still works on a different but related kind, such as night photos or another hospital's.
dot productA score of how well two lists of numbers line up, found by multiplying matching entries and adding the products. It is large when the lists point the same way and zero when they are at right angles.
early exitLetting an AI model stop partway through its layers and answer early when it is already confident, saving computing effort on easy inputs.
eigenvectorA special direction for a transformation: anything pointing that way stays on its line, only longer, shorter or reversed; the amount of stretch is the eigenvalue. Together they reveal the main axes along which data or a system varies.
elastic weight consolidationA way to help a neural network learn a new task without forgetting an old one, by making the settings that mattered most for the old task harder to change, as if tied down with springs.
ELMoA 2018 model whose word representations changed with context, so 'bank' got different numbers in 'river bank' and 'bank loan'. It was a stepping stone between fixed word vectors and BERT.
embedding modelAn AI model whose only job is to turn text (or images, or code) into lists of numbers that capture meaning, so a computer can measure how similar two things are.
encoder-only transformerA transformer that reads a whole text at once, in both directions, to understand it rather than to write new text. Such models power search, sorting documents by topic and spotting names in text.
end-to-end learningTraining one AI system to go straight from raw input, such as camera images, to the final output, such as steering commands, instead of chaining hand-designed steps.
ensemble learningCombining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model.
evasion attackAn attack on an already-trained AI system in which the attacker shapes an input — an email, a file, an image — so the system misjudges it, letting spam or malware slip through.
experience replayStoring an AI agent's past experiences in a memory bank and repeatedly learning from random samples of them, rather than only from what just happened.
fast gradient sign methodA quick way to make an image that fools a neural network: nudge every pixel slightly in whichever direction most increases the network's error.
featureOne piece of information about each example that a model uses, like a house's size or a pixel's color, or a pattern a model has learned to detect inside itself.
feature absorptionA flaw in concept-finding tools where a general concept goes missing in some cases because a more specific concept swallows it — like a 'starts with S' detector staying silent on the word 'snake'.
feature attributionGiving each input to an AI model — each pixel, word or data column — a score for how much it pushed a particular output up or down.
feature extractionTurning raw data such as pixels or sound into a short list of telling numbers — computed by hand-designed rules or by an AI system trained earlier — that a new model can learn from or compare.
feature (interpretability)A concept or property that a neural network represents internally — such as 'is French', 'mentions a bridge' or 'this code has a bug' — usually corresponding to a direction in its internal activity.
feature splittingWhat happens when a concept-finding tool is given more room: a broad concept it found before breaks into several narrower ones, much as 'animal' might split into 'dog', 'cat' and 'bird'.
feature visualizationMaking pictures that show what a part of an image-recognition network is looking for, by repeatedly adjusting an image until that part responds as strongly as possible.
feed-forward networkThe part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge.
few-shot learningGetting an AI system to handle a new task or recognize a new kind of thing from just a handful of examples, the way a person can learn a new word after hearing it once or twice.
fine-tuningTaking an AI model that has already learned a lot and training it a little more on a smaller, focused set of examples so it gets better at one job or behaves a certain way.
Fisher informationA number, or table of numbers, saying how sharply the probability of the data reacts when a model's settings are nudged; the sharper the reaction, the more precisely the data can pin those settings down.
FlashAttentionA faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip.
floating-point formatA way of storing numbers in a computer like scientific notation — a few digits plus a power of two — trading accuracy for range. AI uses ever-smaller formats to compute faster.
flow matchingA way to train generators that learns, for every point between random noise and real data, which direction to move, so that noise can be steered smoothly into new examples.
Fourier transformA way of breaking a signal — a sound, an image, any pattern over time or space — into the simple waves of different frequencies that add up to it, much as a prism splits white light into colors.
FP16A 16-bit way of storing numbers that uses half the memory of the standard 32-bit format. It is fast for AI but can't represent very large or very tiny values.
FP32The standard 32-bit way computers store numbers with fractional parts, keeping about seven significant digits. It was the default for training AI until faster, smaller formats took over.
Kunihiko FukushimaA Japanese engineer who in 1979 invented the neocognitron, a brain-inspired image-recognition network that was the direct ancestor of today's convolutional neural networks.
Gemini RoboticsA family of AI models from Google DeepMind, built on its Gemini models, that let robots understand what they see and what people ask of them, and turn that understanding into physical actions.
generalizationHow well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers.
generative adversarial networkTwo neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people.
global workspace theoryA leading scientific theory holding that we become conscious of something when the brain broadcasts it widely — as if putting it on a stage where every part of the mind can use it.
GloVeA 2014 Stanford method for learning number lists that represent words, built from counts of how often each pair of words appears near each other across a very large body of text.
Ian GoodfellowAn American computer scientist who invented generative adversarial networks, in which two neural networks compete to produce realistic fake images, and who pioneered research on fooling AI systems.
Google BrainGoogle's deep-learning research team from 2011 to 2023, where the transformer — the design behind today's chatbots — and the TensorFlow software were created.
GPGPUUsing a graphics chip for ordinary number-crunching instead of drawing pictures — the trick that, years later, made modern AI training practical.
GPUA computer chip first designed to draw video-game graphics that turned out to be ideal for AI, because it can do thousands of simple calculations at the same time.
Grad-CAMA technique that shows, as a coarse heat map, which regions of an image a vision network relied on when choosing a particular label.
gradientA list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill.
gradient hackingA hypothetical tactic, not yet seen in practice, in which an AI under training picks its answers to steer how training reshapes it, usually to keep its present goals intact.
gradient inversion attackRebuilding someone's private training data — such as photos or text — just from the model updates their device shared during collaborative training.
gradient maskingWhen a defense against attacks only seems to work because it confuses the attacker's tools, not because the model is really any harder to fool.
guided backpropagationA way of producing sharp, detailed pictures of which parts of an image a vision network responded to, by passing only positive signals backward through the network.
Kaiming HeA Chinese computer scientist whose ResNet design, with shortcut connections that let networks grow hundreds of layers deep, is among the most cited research in all of science.
Donald HebbA Canadian psychologist who proposed in 1949 that connections between brain cells strengthen when the cells are active together, an idea behind many learning rules for neural networks.
Hebbian learningThe idea that a connection between two brain cells grows stronger when they are active at the same time — the principle behind the slogan 'cells that fire together wire together'.
HessianA table of 'slopes of slopes' that describes how a surface curves: whether a point is the bottom of a bowl, the top of a hill or a mountain pass.
Geoffrey HintonA British-Canadian scientist whose decades of persistent work on brain-inspired neural networks laid the foundations of modern AI; often called a 'godfather of AI', he now warns publicly about its dangers.
Sepp HochreiterA German computer scientist who explained why early neural networks struggled to learn long sequences and, with Jürgen Schmidhuber, invented the LSTM network that solved the problem.
John HopfieldAn American physicist who showed in 1982 how a network of simple units can store and recall memories, and who shared the 2024 Nobel Prize in Physics for work that enabled machine learning.
Hopfield networkA network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint.
Huber lossA way of measuring prediction errors that treats small errors like squared distance but large ones more gently, so a few wild data points cannot drag the whole model off course.
hybrid architectureAn AI model design that mixes two kinds of building blocks — usually a few standard attention layers among cheaper, memory-saving layers — to keep most of the quality at lower cost on long inputs.
hyperparameterA setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data.
ICLRA major deep-learning conference, held every spring since 2013, where the reviews of each submitted paper are posted online for anyone to read.
image classificationTeaching a computer to say what a picture shows by choosing a label, such as 'cat', 'pneumonia' or 'stop sign', from a fixed list of possibilities.
image segmentationDividing a picture into regions by deciding, pixel by pixel, which object or kind of surface each tiny dot belongs to, like coloring inside the outline of everything.
image-to-image translationTurning one picture into another version of itself — a sketch into a photo, a day scene into night, a horse into a zebra — while keeping its layout.
in-context learningA language model's ability to pick up a new task from instructions or examples placed in its input, on the spot, without any retraining — like a quick study learning a game's rules from one round.
inductive biasThe assumptions built into a learning method that decide how it fills the gaps between the examples it was shown, such as that similar inputs get similar answers. Every learner needs some to cope with new cases.
information bottleneckThe idea that a good internal summary of data keeps exactly the details needed for the task and discards everything else, as a good note-taker does.
instance segmentationFinding each separate object in an image and tracing its exact outline, so that two people standing together get two distinct shapes rather than one blob.
integrated gradientsA way to score how much each input feature contributed to a neural network's output by gradually morphing a blank 'baseline' input into the real one and adding up the network's sensitivity along the way.
Alexey IvakhnenkoA Soviet Ukrainian scientist who from 1965 built some of the first deep, many-layered learning networks, growing them layer by layer from data, decades before modern deep learning.
JacobianA table listing how each output of a process responds to a small change in each input. It describes how a transformation locally stretches, squeezes and turns space.
JAXA free Google library that lets researchers write mathematics in familiar Python and automatically get derivatives and very fast code for graphics chips and Google's AI chips.
joint-embedding predictive architectureA way of training AI on images or video in which the model predicts an abstract summary of a hidden part, rather than every pixel, so it learns what matters and ignores unpredictable detail.
John JumperAn American scientist who led Google DeepMind's AlphaFold project, which predicts the 3D shapes of proteins, and shared the 2024 Nobel Prize in Chemistry.
Andrej KarpathyA Slovak-Canadian AI researcher and teacher who co-founded OpenAI, led Tesla's self-driving vision team, and coined popular phrases such as 'vibe coding'.
Koray KavukcuogluA Turkish-born computer scientist who co-authored many of DeepMind's landmark papers and in 2026 took charge of Google DeepMind as its senior vice president.
KerasA free, beginner-friendly Python library for building neural networks from ready-made layers in a few lines of code, while a larger toolkit underneath does the heavy computing.
kernel fusionMerging several small calculation steps into one, so a chip finishes them in a single pass instead of saving and reloading data in between, like cooking at the counter without returning ingredients to the fridge after each step.
Diederik KingmaA Dutch computer scientist who co-invented the variational autoencoder and the Adam optimizer, two of the most widely used tools for training neural networks.
knowledge distillationTraining a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.
knowledge graph embeddingA way of turning the things and relations in a knowledge graph into lists of numbers, so a computer can guess missing facts from patterns, such as predicting where someone was born.
Teuvo KohonenA Finnish engineer who invented the self-organizing map, a neural network that arranges complex data on a simple grid so that similar items end up next to each other.
Alex KrizhevskyA Canadian computer scientist who built AlexNet, the image-recognition network that was trained on gaming graphics chips and whose 2012 contest win launched the deep learning boom.
label noiseMistakes in the answers attached to training or test examples — a dog photo marked 'cat' — which can mislead what a model learns and distort how good it seems.
large language modelA computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code.
latent diffusion modelA diffusion image generator that works on a shrunken, compressed version of each picture instead of the full-size pixels, which makes it far cheaper to train and run.
latent reasoningGetting an AI to do its step-by-step thinking inside its internal numbers rather than writing it out in words — potentially faster and more flexible, but much harder for people to inspect.
layer-wise relevance propagationA method that traces a neural network's decision backward, layer by layer, handing each part of the input its share of the credit until every pixel or word has a relevance score.
Yann LeCunA French-American computer scientist who pioneered the neural networks that recognize images, led Meta's AI research for over a decade, and argues that chatbots alone will not reach human-level AI.
LeNetA family of early image-recognition neural networks built at Bell Labs from 1989 that learned to read handwritten digits. A version read huge numbers of bank checks, and its design underlies today's image models.
linear algebraThe branch of mathematics about lists and grids of numbers and the even, straight-line transformations between them. It is the everyday language in which neural networks are written and computed.
linear attentionA family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words.
linear separabilityWhen two groups of points can be perfectly divided by a single straight line or, in more dimensions, a flat surface. Many simple learning methods only fully succeed in this case.
linear transformationA rule for moving every point in a space that keeps straight lines straight and the origin in place: rotating, stretching, reflecting or squashing everything evenly. Every grid of numbers describes one.
Seppo LinnainmaaA Finnish computer scientist who in 1970 first published the method for computing all of a program's derivatives in one backward pass, the mathematical core of backpropagation.
local learning coefficientA number measuring how complex a trained neural network effectively is around its current settings — usually far less than its raw parameter count suggests — used to spot changes as the network learns.
local minimumA spot lower than everything immediately around it, like the bottom of a small valley, even though a deeper valley may exist somewhere else.
local SGDA way to train on many machines that talk to each other only occasionally: each trains its own copy for a while, then they average their copies and continue, cutting the amount of communication needed.
loss curveA graph of a model's error as training goes on. A healthy curve drops steeply and then keeps sloping gently downward; bumps, jumps or flat stretches can warn engineers that something is wrong.
loss of plasticityThe gradual loss of a neural network's ability to learn new things after long training on changing tasks, as parts of the network go inactive or become stuck.
loss spikeA sudden jump in a model's error partway through training. Sometimes the model recovers on its own; sometimes the run falls apart and engineers must rewind to an earlier saved copy.
machine learningA way of building software that learns from examples instead of following rules written by a programmer. Shown thousands of labeled photos, for instance, a program can learn to recognize cats.
MambaA 2023 alternative to the transformer design that processes text with a continually updated summary which decides what to keep or forget, letting it handle long sequences quickly.
manifold hypothesisThe idea that real-world data, such as photos of faces, fills only a thin, smoothly curved sliver of the vast space of all possible data — which is why learning from examples works at all.
Gary MarcusAn American cognitive scientist and leading critic of deep learning, who argues that today's AI cannot reason reliably and needs to be combined with symbolic methods.
Mark I PerceptronA room-sized machine built around 1960 that learned to recognize simple shapes and letters through a camera made of 400 light sensors. It was one of the first machines built to learn from examples.
masked autoencoderA way to teach a vision model without labels: hide most of an image's tiles and train the model to fill in the missing parts, so that it learns what images are made of.
matrixA rectangular grid of numbers arranged in rows and columns. Most of what a neural network has learned is stored in large grids like this.
matrix multiplicationThe rule for combining two grids of numbers into a new grid by repeatedly multiplying rows by columns. It is the single calculation AI chips spend most of their time doing.
Matryoshka embeddingAn embedding built like a set of Russian nesting dolls: its first few numbers already give a rough summary, so you can cut it short to save space and still search reasonably well.
maximal update parametrizationA recipe for scaling a neural network's starting values and learning speeds with its size so that settings tuned on a small, cheap model work unchanged on a huge one.
Warren McCullochAn American brain scientist who, with Walter Pitts, wrote the 1943 paper that first described nerve cells as simple logic units, the seed of today's artificial neural networks.
McCulloch–Pitts neuronThe first mathematical model of a brain cell as a simple on–off switch: it fires if enough of its inputs are on and no blocking input is. Networks of them can carry out logic.
mechanistic interpretabilityA branch of AI research that tries to reverse-engineer a neural network — working out, piece by piece, which internal parts represent which concepts and how they combine to produce behavior.
Meta AIThe artificial intelligence research group of Meta, the company behind Facebook and Instagram, known for the PyTorch software and the Llama models. It is also the name of Meta's chatbot.
meta-learningTraining a system across many different tasks so that it gets better at learning itself, letting it pick up a new task quickly from only a few examples.
Microsoft ResearchMicrosoft's research division, with laboratories on several continents, whose scientists have produced influential work on speech, vision and language AI.
Tomas MikolovA Czech computer scientist who created word2vec, a fast way to turn words into lists of numbers that capture their meaning, and pioneered language models built from recurrent networks.
MilaMontreal's deep-learning hub, where professors and students from several Quebec universities work side by side. It grew from the lab Yoshua Bengio started in 1993 and is among the world's largest academic AI centers.
mixture of expertsA way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
Volodymyr MnihA DeepMind researcher who led the 2013–2015 work in which one neural network learned to play dozens of Atari video games from screen pixels, launching deep reinforcement learning.
mode collapseWhen a generator gets stuck producing only a few kinds of output — the same face, the same joke — instead of the full variety it was meant to learn.
model capacityHow much a model is able to learn: the variety and complexity of patterns it can represent or memorize. Too little and it misses real patterns; very large models can even memorize pure noise.
model compressionThe family of techniques for making AI models smaller, faster or cheaper to run while keeping almost all of their accuracy.
model inversionUsing a trained AI model's answers to work backwards to private information it learned from — for example, rebuilding a recognizable picture of someone's face from a face-recognition system.
model mergingCombining two or more trained AI models into one by blending their internal numbers directly, so the result shares their skills without any further training.
model soupA single model made by averaging the internal numbers of several copies of a model that were fine-tuned in slightly different ways, often doing better than any one of them.
model-agnostic meta-learningA training method that searches for a starting point for a model's internal settings from which it can learn any new task of a given kind with just a few small adjustments.
monosemanticityThe property of an internal unit of an AI model that stands for just one understandable thing, which makes it far easier to interpret than a unit that mixes several.
multi-head attentionRunning several attention processes side by side, each free to track a different kind of relationship between words — grammar, topic, who did what — and then combining their results.
multi-task learningTraining one model to do several related jobs at once, such as spotting objects and judging distance in the same photo, so that practice on each job can sharpen the others.
mutual informationHow much knowing one thing tells you about another — for example, how much a word reveals about the topic of a document. It is zero when the two are unrelated.
neocognitronA 1980 neural network for recognizing handwritten characters, modeled on the brain's visual system. Its stacked layers of pattern detectors were the blueprint for the networks used in modern image recognition.
network dissectionA method for finding out what individual units inside an image-recognition network respond to, such as dogs, windows, grass or stripes, by comparing where each unit lights up with pictures labeled by people.
neural networkA kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain.
neural radiance fieldA way of capturing a real scene from a set of photos as a small neural network that can then show the scene from any new viewpoint, as if moving a camera through it.
NeurIPSThe largest yearly scientific conference on artificial intelligence and machine learning, held each December, where many of the field's landmark papers were first presented.
neuroevolutionTraining or designing neural networks by simulated evolution — breeding and mutating whole networks and keeping the best performers — instead of, or alongside, the usual gradient-based training.
neuromorphic computingComputer chips designed to work more like brains, with artificial neurons that send brief spikes only when needed, aiming to do AI tasks with far less energy.
neuro-symbolic AIAI that combines neural networks, which learn patterns from data, with symbolic methods, which follow explicit rules and logic, aiming for systems that both learn flexibly and reason reliably.
Andrew NgA British-born American computer scientist whose online courses have taught machine learning to millions; he also co-founded Google's deep learning team, Google Brain, and the online-course company Coursera.
N:M sparsityA tidy pattern of zeros inside an AI model, such as two zeros in every group of four numbers, that special chip hardware can skip to run the model faster.
noise scheduleThe plan for how much static is mixed into training examples at each step of a diffusion model, from almost none to total noise; its shape affects quality and speed.
normA way of measuring the size, or length, of a list of numbers — for example the straight-line distance from the origin to the point the list describes.
normalizing flowA generator built from a chain of reversible steps that reshape simple random noise into realistic data; because every step can be undone, it can also calculate exactly how likely any example is.
numerical stabilityWhether a computer calculation stays accurate despite the tiny rounding errors in every step, instead of blowing up into nonsense such as infinities.
object detectionFinding every object of interest in a picture or video frame, saying what each one is and drawing a box around it, for example all the cars, cyclists and pedestrians in a street scene.
occlusion sensitivityFinding out which parts of an image matter to a model by covering them up one patch at a time and watching how much the model's answer changes.
Chris OlahA Canadian researcher who pioneered ways of looking inside neural networks to see what they have learned, and a co-founder of the AI company Anthropic.
one-hot encodingTurning a category such as 'red', 'green' or 'blue' into a row of zeros with a single one marking which category it is, so that a model can use it as numbers.
ONNXA shared file format for trained AI models, so that a model built with one toolkit can be moved to and run by another, much as a PDF opens on any computer.
open-vocabulary detectionObject detection where you type what to look for, such as 'red umbrella' or 'forklift', and the system finds it, even if it was never trained on that exact category.
optimizationFinding the best possible choice according to some score, such as the lowest error or the highest reward. Training an AI is optimization: adjusting its settings until its mistakes are as small as possible.
orthogonalityThe mathematical version of being at right angles. Two directions are orthogonal when moving along one tells you nothing about the other, like north and east on a map.
overfittingWhen a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
overthinkingWhen an AI reasoning model spends far more thinking than a problem needs — pages of deliberation for a simple sum — wasting time and money and sometimes talking itself out of a right answer.
panoptic segmentationA complete labeling of an image in which every pixel gets a category, such as road or sky, and every countable object, such as each car or person, also gets its own identity.
Seymour PapertA South African-born mathematician who co-wrote 'Perceptrons', a critique of early neural networks, with Marvin Minsky, and invented the children's programming language Logo.
Parallel Distributed ProcessingA two-volume 1986 book by a group of psychologists and computer scientists that revived interest in brain-like neural networks and made the training method behind modern AI widely known.
parameter countHow many adjustable numbers an AI model contains — the usual measure of its size. Modern language models have billions to trillions of them.
parameter decompositionAn approach that tries to understand a neural network by splitting its weights — rather than its internal activity — into many simple pieces, each responsible for one mechanism.
partial derivativeHow much a quantity changes when you nudge just one of its many inputs while holding all the others fixed.
patch embeddingThe first step of a vision transformer: cutting an image into small square tiles and turning each tile into a list of numbers that the model can process like a word.
Perceptrons (book)A 1969 book by Marvin Minsky and Seymour Papert proving that simple one-layer neural networks cannot learn certain patterns. It is widely blamed for a decade-long slump in neural-network research.
perceptual lossA way of judging how different two images look to a person by comparing what a trained image-recognition network 'sees' in each, instead of comparing pixels one by one.
π0A general-purpose AI model for controlling robots, released in 2024 by the start-up Physical Intelligence, that sees through cameras, follows typed or spoken instructions and moves robot arms deftly enough to fold laundry.
Walter PittsA largely self-taught American logician who, at about 20 and without a degree, co-invented with Warren McCulloch a simple logical model of a brain cell, the starting point of neural networks.
platonic representation hypothesisThe idea that bigger, more capable AI models, even ones trained on different things such as images and text, end up organizing information in increasingly similar ways, as if all were mapping the same underlying world.
polysemanticityWhen a single artificial neuron responds to several unrelated things — say, bird beaks, bicycle wheels and a particular typeface — so that no single label describes what it does.
positional encodingExtra information added to each word so a transformer knows where it sits in the text. Without it, 'dog bites man' and 'man bites dog' would look the same to the model.
post-training quantizationShrinking an already-trained AI model by storing its numbers in fewer bits, without training it again; a small sample of example inputs is usually enough to fine-tune the rounding.
predictive processingThe idea that the brain is a prediction machine: it constantly guesses what its senses will report next and learns mainly from its mistakes, so what we perceive is largely its best guess.
privileged basisA property of a layer in a neural network where the individual numbered slots, not just blends of them, carry special meaning, because something in the design treats each slot separately.
projected gradient descent attackA stronger version of the quick pixel-nudging attack: take many small steps that each raise the model's error, pulling the image back after every step so the total change stays tiny.
prototypical part networkAn image classifier that explains each decision by pointing to parts of the picture that resemble typical examples it learned — this part looks like that part of a known bird.
pruningCutting away the parts of an AI model that contribute least, as a gardener trims branches, so the model becomes smaller or faster while still working nearly as well.
PyTorchThe most widely used free software toolkit for building and training neural networks, especially in research; programmers write ordinary Python and it handles the calculus and the graphics chips.
QK-normA small change to how a language model pays attention — rescaling two sets of internal numbers before comparing them — that prevents a common way large models' training goes haywire.
quantizationStoring an AI model's numbers with fewer digits, like rounding prices to the nearest dollar, so the model takes less memory and runs faster, usually with only a small loss of accuracy.
quantization-aware trainingTraining or fine-tuning an AI model while simulating the low-precision numbers it will use later, so that it learns to work well despite the rounding.
query, key and valueThe three roles each word plays inside attention: the query is what it is looking for, the key is what it can be found by, and the value is the information it hands over when chosen.
randomized smoothingA way to make a classifier provably harder to fool: ask it about many noisy copies of an input and take a majority vote, which small deliberate changes cannot easily flip.
R-CNNA family of object detectors that first pick out image regions likely to contain objects, then use a neural network to label each region and tighten its box.
rectified flowA generator trained to move random noise toward real data along straight lines, which can then be straightened further so that a picture comes out in one or a few steps.
regularizationAny technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
reparameterization trickA trick that rewrites a random draw as fixed numbers plus separate, outside randomness, so that a neural network can still learn through a step that involves chance.
residual streamThe running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.
reversal curseA quirk of language models: having learned that 'A is B', they often cannot answer the reverse question — they may know who a celebrity's mother is yet fail to say who her son is.
Frank RosenblattAn American psychologist who invented the perceptron, the first neural network that learned from examples, and built a machine that learned to recognize simple images.
rotary positional embeddingA way of telling a transformer where each word is by rotating parts of its internal numbers by an angle that grows with position, so the model can read off how far apart two words are.
routerIn a mixture-of-experts model, the small component that decides which specialist sub-networks should handle each word, like a dispatcher assigning jobs to workers.
David RumelhartAn American psychologist who, with Geoffrey Hinton and Ronald Williams, showed in 1986 how to train multilayer neural networks, launching the modern revival of the field.
RWKVAn open-source language model design that works like an old-style recurrent network when writing text, using constant memory, yet can be trained quickly in parallel like a transformer.
saddle pointA spot on a surface that is a low point in one direction but a high point in another, like the middle of a horse's saddle or a mountain pass.
saliency mapA heat map laid over an image or text that highlights which parts most influenced an AI model's output — or, in vision science, which parts of a scene would most catch a viewer's eye.
scalarA single number, such as 3.7 or −2, as opposed to a list or grid of numbers. A model's error score and its learning rate are both scalars.
scaled dot-product attentionThe standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information.
Jürgen SchmidhuberA German computer scientist who co-invented the LSTM, a neural network with memory that powered much of speech recognition and translation in the 2010s, and who argues that early pioneers go under-credited.
score-based generative modelA generator that learns, for any noisy example, which direction would make it look more like real data — then starts from pure noise and follows those directions step by step.
Segment Anything ModelA family of AI models from Meta that trace the exact outline of whatever you point to in a photo or video, whether by clicking on it, drawing a box around it or, in newer versions, naming it.
Terry SejnowskiAn American neuroscientist who co-invented the Boltzmann machine with Geoffrey Hinton, built an early network that learned to read aloud, and helped found the NeurIPS conference community.
self-attentionAttention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to.
self-supervised learningA way for AI to learn from huge amounts of unlabeled data by setting itself fill-in-the-blank puzzles, such as guessing a hidden word or a missing patch of an image.
semantic segmentationColoring every pixel of an image by what it shows, such as road, sky, person or car, without telling one person or car apart from another.
Sentence-BERTA way of turning a whole sentence into a list of numbers so that sentences with similar meanings get similar numbers, which makes it fast to search millions of texts by meaning rather than by exact words.
Noam ShazeerAn American computer scientist who co-invented the transformer and several techniques found in nearly every large language model, co-founded Character.AI, and later returned to Google to co-lead Gemini.
shortcut learningWhen an AI system gets right answers for the wrong reasons — noticing which hospital an X-ray came from rather than signs of disease — so it fails once that accidental clue disappears.
singular learning theoryA mathematical theory of learning for models like neural networks, whose many redundant settings break standard statistics; it explains how such models generalize and how their learning unfolds.
singular value decompositionA way of rewriting any grid of numbers as three simple moves in turn — turn, stretch along perpendicular axes, turn again — which reveals the directions that matter most in the data.
SmoothGradA trick for making explanation heat maps less speckled: compute the map for many noisy copies of the input and average them, so random flicker cancels out.
Software 2.0Andrej Karpathy's 2017 name for software whose behavior is learned from examples by a neural network instead of being written line by line by programmers.
Jascha Sohl-DicksteinAn American scientist who in 2015, with colleagues, introduced diffusion models, the technique behind most modern AI image and video generators, by borrowing ideas from physics.
sound event detectionTeaching computers to recognize everyday sounds in a recording, such as a dog barking, glass breaking or a siren, and to say exactly when each one starts and stops.
sparse autoencoderA small add-on network trained to break an AI model's tangled internal signals into a much larger set of simpler pieces, most of which can be labeled with an understandable concept.
sparse upcyclingRecycling an already-trained AI model into a larger 'team of specialists' model by copying parts of it, so the new model starts from what the old one learned instead of from scratch.
sparsityThe property of being mostly zeros or mostly switched off. AI systems exploit it by skipping the empty parts, so they need less memory and computation.
special tokenA reserved symbol in an AI model's vocabulary that is not ordinary text but a signal — 'start of message', 'end of answer', 'begin thinking' — used to structure what the model reads and writes.
stability–plasticity dilemmaThe basic tension any learning system faces: it must change enough to learn new things, yet stay stable enough not to lose what it already knows.
state space modelA type of sequence model that reads text like a running tally, updating a fixed-size internal summary at each step, so its memory and cost do not balloon as texts get longer.
statistical learning theoryThe mathematics of when and why learning from examples works: how much data is needed, and how well a model trained on some examples will do on new ones.
StyleGANAn image-generating network from NVIDIA, famous for photorealistic faces of people who do not exist, that sets coarse traits such as head angle and face shape separately from fine details such as coloring and skin texture.
superpositionThe way a neural network squeezes more concepts into its internal activity than it has neurons, by letting them overlap — a main reason its insides are hard to read.
Ilya SutskeverA Russian-born Israeli-Canadian computer scientist who co-built the AlexNet image recognizer, helped found OpenAI and lead its research until 2024, and now heads the startup Safe Superintelligence Inc.
systematicityThe way thinking and language come in connected packages: if you can understand 'the dog chases the cat', you can also understand 'the cat chases the dog'. Whether neural networks manage this has long been debated.
systolic arrayA grid of tiny calculators that pass numbers to their neighbors in rhythm, like a heartbeat, so each value is reused many times. AI chips use it to multiply large tables of numbers efficiently.
tabular foundation modelA neural network pretrained on millions of made-up tables so that, given a new spreadsheet of labeled examples, it can predict the missing answers at once, without being retrained for that table.
target networkA slowly updated copy of an AI agent's learning network, used to compute the goals it learns toward, so those goals do not shift every time the network changes.
task arithmeticEditing an AI model by adding or subtracting the changes that fine-tuning made to it, as if skills were arrows that can be added together or reversed.
Taylor seriesA way of approximating a complicated curve near a chosen point by adding up simple pieces: a constant, a straight-line slope, a bend, and so on.
tensorA block of numbers with any number of dimensions: a single number, a list, a grid, or a stack of grids. AI software stores and processes all its data this way.
tensor coreA small, specialized part of an AI chip that does one job — multiplying small blocks of numbers together — far faster than ordinary circuits can.
TensorFlowGoogle's free software toolkit for building, training and running machine-learning models; it led the field after 2015 and still runs many models inside phones, browsers and company systems.
testing with concept activation vectorsA method for asking whether a human idea, such as 'wheels', matters to an image model's decision, such as 'car', by finding the direction for that idea inside the model and measuring its effect.
test-time trainingLetting an AI model briefly update itself on each new problem it is given, learning from that problem's own clues before answering, instead of keeping its knowledge frozen after training.
text embeddingA list of numbers that captures what a piece of text means, so that texts with similar meanings get similar lists. Computers use them to search, sort and group text by meaning.
TheanoOne of the first popular free software tools for deep learning, built at a Montreal university lab; development ended in 2017, but it shaped the toolkits that replaced it.
time series foundation modelA large AI model pretrained on huge numbers of time series, such as sales, traffic and sensor readings, that can forecast a new series it has never seen without being trained on it.
toy modelA deliberately tiny, simplified AI model trained on an artificial task, small enough for researchers to understand completely and use to test ideas about how big models work.
TPUGoogle's own family of chips built specifically for AI, used to train and run its Gemini models and rented to other companies through its cloud.
trainingThe process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better.
training instabilityThe tendency of very large training runs to go wrong — the error suddenly jumping or blowing up — which can waste weeks of expensive computing unless engineers prevent or repair it.
transcoderA stand-in for one part of an AI model that does the same job using a large set of simple, understandable pieces, so researchers can see what that part computes.
transformerThe design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
transformer blockThe repeating building unit of a transformer. A model is a stack of dozens of identical blocks, each pairing an attention step that mixes information between words with a step that processes each word.
TransformersA free software library from Hugging Face that lets programmers download and use thousands of ready-trained AI models for text, images and sound with a few lines of code.
underfittingWhen a model is too simple, or trained too little, to capture the real pattern in the data, so it does poorly even on the examples it learned from.
unembeddingThe last step of a language model, which turns its internal state into a score for every token in its vocabulary; the next token is then chosen from those scores.
universal adversarial perturbationA single faint pattern that, added to almost any image, makes a neural network misclassify it — one key that opens many locks.
universality hypothesisThe idea that different AI models trained on similar tasks tend to develop the same internal concepts and mechanisms, so what is learned about one model may carry over to others.
Jakob UszkoreitA German computer scientist, one of the eight authors of the transformer paper, credited with first proposing to build language models on attention alone.
varianceA number that grows the more widely something's possible values scatter away from its average, and is zero if it never varies. Its square root, the standard deviation, uses the same units as the values themselves.
variational autoencoderA neural network that learns to squeeze data into a compact, slightly fuzzy code and rebuild it, arranged so that decoding random codes produces new, plausible examples.
Ashish VaswaniAn Indian-born computer scientist listed first on 'Attention Is All You Need', the 2017 paper that introduced the transformer, the design behind today's language models.
vectorAn ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points.
Vector InstituteToronto's publicly and privately funded AI institute, set up in 2017 with Geoffrey Hinton as an adviser, which supports university researchers and helps companies, hospitals and government put their work to use.
Oriol VinyalsA Spanish computer scientist who co-created key neural methods for translation and image captioning, led DeepMind's StarCraft-playing AlphaStar, co-led Google's Gemini models, and in 2026 co-founded the startup Discovery Loop.
vision transformerA way of applying the transformer, the design behind chatbots, to pictures: the image is cut into small square tiles that the model reads like words in a sentence.
VQGANAn image compressor that turns a picture into a small grid of codebook 'tiles' while keeping it sharp, used as the vocabulary of many image generators and of an early wave of AI art.
VQ-VAEA network that compresses an image or sound into a short list of entries from a learned 'codebook' — like describing a picture with a fixed set of tiles — which other models can then generate.
warmup-stable-decay scheduleA plan for how fast a model learns during training: speed up briefly, hold steady for most of the run, then slow down sharply at the end — which lets labs stop or extend training without starting over.
Wasserstein GANA version of the forger-versus-detective image generator that measures how far its fakes are from real data in a smoother way, making training steadier and easier to monitor.
wav2vecA family of Meta speech models that learn the sounds of language from thousands of hours of unlabeled recordings, so that only minutes of transcribed speech are needed to build a recognizer.
WaveNetA DeepMind model that generates speech and other sounds one tiny audio sample at a time, producing voices far more natural than earlier computer speech.
weightsThe learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself.
Paul WerbosAn American scientist whose 1974 doctoral thesis set out backpropagation, the method of passing errors backward that now trains nearly all neural networks.
white-box attackAn attack in which the attacker can see everything about an AI model, including its internal settings, and uses that knowledge to find its weak spots.
Bernard WidrowAn American electrical engineer who in 1960 built ADALINE, an early trainable neural unit, and invented a learning rule that adjusts weights to reduce errors, still at the heart of adaptive filters.
word embeddingA list of numbers standing for a word, learned so that words used in similar ways get similar lists. It lets a computer measure that 'cat' is closer to 'dog' than to 'car'.
word2vecA 2013 Google technique that learns a list of numbers for each word by training a very small network to predict which words appear near it across billions of words of text.
X is all you needA joking title formula copied from a famous 2017 AI paper, used to claim that one simple ingredient is enough to solve a problem; hundreds of research papers now use it.
z-lossA small extra penalty added during training that stops the raw scores a language model produces before choosing a word from growing huge, which helps keep very large training runs from blowing up.