forward pass
In plain wordsThe step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction.
The computation of a neural network's outputs from its inputs by evaluating each layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage. in order, each consuming the previous layer's activationsactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment.. During training the forward pass also computes the lossloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down. and stores, or arranges to recompute, the intermediate values the backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change. will need, so it determines activation memory, which grows with batch sizebatch size, n.: How many training examples a model looks at before each adjustment of its settings. Bigger batches use hardware efficiently but bring diminishing returns; smaller ones are noisier but cheaper per step., sequence length and depth. At inferenceinference, n.: Drawing a conclusion from evidence. In today's AI it most often means running an already-trained model to get an answer, as opposed to training it. time a forward pass is all that runs; for an autoregressive large language modellarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., each new token requires one, split in practice into prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer. of the prompt and token-by-token decodingdecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far. that reuses a KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.. Forward-pass cost underlies common compute estimates: a dense transformer needs about 2N floating-point operations per token for N parameters, and training (forward plus backward) about 6N per token.
The forward pass fits in memory, but storing activations for the backward pass does not.
Origin
A descriptive term from the backpropagation literature of the 1980s, contrasting the forward flow of activity from input to output with the backward flow of error signals. Deep-learning libraries made 'forward' the name of the method that implements a layer's computation.
Related
- backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change.
- backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction.
- computational graphcomputational graph, n.: A map of a calculation drawn as boxes and arrows: each box is a simple operation and each arrow carries a number to the next step. AI software uses it to trace errors backward.
- activationactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment.
- prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer.
- training steptraining step, n.: One round of learning: the model makes predictions on a small batch of examples, measures its error and adjusts its settings once. Training runs consist of thousands to millions of steps.
- backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change.
Mentioned in
backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction., FLOPFLOP, n.: A single arithmetic step, such as multiplying or adding two numbers. Counting them measures how much computing went into an AI model, and laws use such counts to decide which models face extra rules., trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better., ablationablation, n.: Removing or switching off a part of an AI system to see what changes — whether an ingredient of an experiment's design or a single unit inside a trained network., activationactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., activation steeringactivation steering, n.: Changing an AI model's behavior by adjusting the internal numbers it computes while it runs — like turning a hidden dial — instead of rewording the prompt or retraining the model., backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change., computational graphcomputational graph, n.: A map of a calculation drawn as boxes and arrows: each box is a simple operation and each arrow carries a number to the next step. AI software uses it to trace errors backward., cross-encodercross-encoder, n.: An AI model that reads a question and a candidate passage together, side by side, and outputs one relevance score. Accurate but slow, so it is used to reorder a short list., decode phasedecode phase, n.: The second stage of an AI's reply, in which it writes the answer one word-piece at a time, each step depending on everything written so far., feedforward neural networkfeedforward neural network, n.: A neural network laid out like an assembly line: each stage hands its result only to later stages, never back, so every answer depends on the current input alone and nothing is remembered between inputs., generative adversarial networkgenerative adversarial network, n.: Two neural networks trained as rivals: one forges fake images, the other tries to spot the fakes. As both improve, the forger's output becomes realistic enough to fool people., gradient accumulationgradient accumulation, n.: A way to train with large batches on limited hardware: the computer processes several small batches one after another, adds up what each says, and only then makes a single update., gradient checkpointinggradient checkpointing, n.: A memory-saving trick: instead of storing every intermediate result needed for learning, the computer keeps only a few and recalculates the rest when needed, trading extra computing time for memory., mixed-precision trainingmixed-precision training, n.: Training a model with shorter, less exact numbers for most of the arithmetic and full-length numbers only where accuracy really matters, which saves a great deal of memory and lets modern chips train far faster., prefillprefill, n.: The first stage of an AI's reply, in which it reads your whole prompt in one go before it starts writing. Long prompts make this stage take longer., speculative decodingspeculative decoding, n.: A speed-up trick in which a small, fast model guesses the next few words and the big model checks all the guesses at once, keeping the right ones: faster answers, same quality., steeringsteering, n.: Guiding what an AI model does — its topic, tone or behavior — while it runs, by changing its instructions, its internal signals or how its next word is chosen, rather than by retraining it., teacher forcingteacher forcing, n.: Training a model that writes one word at a time by always feeding it the correct previous words, rather than its own possibly mistaken guesses, while it learns to predict the next one., training computetraining compute, n.: The total number of arithmetic operations performed to train a model. It is the standard yardstick for the size of an AI training effort, and some laws use it to decide which models face extra rules., training looptraining loop, n.: The repeating cycle at the heart of training software: grab some examples, let the model guess, measure the error, adjust the model, and do it again — often millions of times., training steptraining step, n.: One round of learning: the model makes predictions on a small batch of examples, measures its error and adjusts its settings once. Training runs consist of thousands to millions of steps., activation additionactivation addition, n.: A simple way to steer a language model: run two contrasting prompts such as 'Love' and 'Hate', take the difference between the model's internal states, and add it while the model writes., model FLOPs utilizationmodel FLOPs utilization, n.: The share of a computer's top calculating speed that actually goes into useful AI training work. Real systems often reach only around a third to a half.