loss function
In plain wordsA score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down.
A function that scores a model's output against the desired target, returning a single number that grows with the size of the mistake; training adjusts the model to make the average of this number over the training datatraining data, n.: The examples an AI system learns from. Whatever patterns, skills, gaps and biases are in the training data tend to show up in the finished system. as small as possible, which is empirical risk minimizationempirical risk minimization, n.: The basic recipe of most machine learning: from the models on offer, pick the one with the smallest errors on the training examples, trusting that score as a guide to how it will do on new data., usually carried out by gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.. The choice of loss defines what counts as a mistake and how heavily it is penalized: mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better. for regression, cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text. (log loss) for classification and next-token predictionnext-token prediction, n.: The simple task most AI chatbots were trained on: given some text, guess the next small piece of it. Repeating that guess, piece by piece, produces whole answers., hinge losshinge loss, n.: A way of scoring a classifier's mistakes that charges nothing for answers that are right with room to spare, and charges more and more the further an answer falls on the wrong side. for margin classifiers, contrastive and ranking losses for embeddings and retrieval. Because gradient-based training needs derivatives, the loss is often a smooth surrogate for the quantity people actually care about, such as accuracy, BLEU or task success. The full training objective may add regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data. terms or auxiliary losses (for example a load-balancing loss in a mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.), and in reinforcement learningreinforcement learning, n.: A way of teaching a computer program by trial and error: it tries actions, earns points for good results and penalties for bad ones, and gradually learns which choices pay off over time. and preference tuning the loss is built from rewards or comparisons rather than labels.
Validation accuracy stalled even though the training loss kept falling.
Origin
The idea goes back to the treatment of observational errors by Pierre-Simon Laplace, who favored absolute deviations, and by Adrien-Marie Legendre and Carl Friedrich Gauss, whose method of least squares (1805–1809) minimized squared ones. Abraham Wald made the loss function — which he first called a 'weight function' (1939) — a central object of statistical decision theory; machine learning inherited 'loss' from statistics and 'cost function' from optimization and control.
Usage note
'Loss' and 'cost' are used interchangeably, though some authors reserve loss for one example and cost for the dataset average; objective functionobjective function, n.: The score that an optimization process tries to push as low, or as high, as possible. For an AI, it defines what 'doing well' means during training. is the broader term and includes quantities to be maximized. A falling training loss does not by itself mean better generalizationgeneralization, n.: How well a model copes with new examples it has never seen, rather than just memorizing its training data, like a student who understands the method, not just the answers..
Related
- gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down.
- empirical risk minimizationempirical risk minimization, n.: The basic recipe of most machine learning: from the models on offer, pick the one with the smallest errors on the training examples, trusting that score as a guide to how it will do on new data.
- cross-entropycross-entropy, n.: A score of how surprised a model's predictions are by what actually happens; lower is better. Language models are trained mainly by driving this number down on real text.
- mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better.
- objective functionobjective function, n.: The score that an optimization process tries to push as low, or as high, as possible. For an AI, it defines what 'doing well' means during training.
- loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys.
- regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data.
- reward functionreward function, n.: The rule that decides how many points an AI agent earns for each situation and action — in effect, the written-down definition of the task it is being trained to do.
Mentioned in
backpropagationbackpropagation, n.: The method neural networks use to learn from mistakes: it works backward from the error to figure out how much each internal connection contributed, so every connection can be adjusted in the right direction., gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down., neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain., optimizationoptimization, n.: Finding the best possible choice according to some score, such as the lowest error or the highest reward. Training an AI is optimization: adjusting its settings until its mistakes are as small as possible., regressionregression, n.: Predicting a number rather than a category — a house's price, tomorrow's temperature, a patient's blood pressure — by learning from past examples where the answer is known., trainingtraining, n.: The process by which an AI model learns: it is shown huge numbers of examples and its internal settings are adjusted, bit by bit, until its answers get better., backward passbackward pass, n.: The step in training where the network works backward from its error, layer by layer, to calculate how each internal setting should change., chain rulechain rule, n.: A rule for working out how a change ripples through a chain of steps by combining the effect at each step. It is how AI training traces each error back through every layer., derivativederivative, n.: How fast something changes as one of its inputs changes — the steepness of a curve at a point. Training an AI means repeatedly following such slopes downhill., empirical risk minimizationempirical risk minimization, n.: The basic recipe of most machine learning: from the models on offer, pick the one with the smallest errors on the training examples, trusting that score as a guide to how it will do on new data., expected valueexpected value, n.: The long-run average you would get if a chancy situation could be repeated many times — 3.5 for a fair die, even though no single roll shows 3.5., experiment trackingexperiment tracking, n.: Keeping an automatic logbook of every model-training attempt — settings, data, code and results — so a team can compare attempts, repeat the good ones and see what changed., focal lossfocal loss, n.: A way of scoring a model's mistakes during training that pays less attention to examples it already gets right easily, so that learning concentrates on the hard and rare cases., forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction., gradientgradient, n.: A list of slopes showing which way, and how steeply, a quantity rises as each of its inputs changes. Training nudges an AI's settings in the opposite direction, downhill., gradient boostinggradient boosting, n.: A method that builds a strong predictor by adding small decision trees one at a time, each trained to correct the errors that remain after all the trees before it., KerasKeras, prop. n.: A free, beginner-friendly Python library for building neural networks from ready-made layers in a few lines of code, while a larger toolkit underneath does the heavy computing., law of large numberslaw of large numbers, n.: The mathematical rule that averages of chance events steady out over many repetitions: flip a fair coin a few times and the share of heads can be far from one-half, but over thousands of flips it gets very close., learninglearning, n.: Getting better at something through experience. For a machine, it means changing its internal settings based on data so that it performs a task better than before., loss curveloss curve, n.: A graph of a model's error as training goes on. A healthy curve drops steeply and then keeps sloping gently downward; bumps, jumps or flat stretches can warn engineers that something is wrong., loss landscapeloss landscape, n.: A way of picturing training as a journey across hilly terrain, where position stands for the model's settings and height for its error. Training searches for low valleys., mean absolute errormean absolute error, n.: A way of scoring numerical predictions: average how far off each prediction is, ignoring direction, so the score reads as 'typically off by this much'. Lower is better., mean squared errormean squared error, n.: A way of scoring numerical predictions: square each prediction's error, then average them, so big misses count much more than small ones. Lower is better., objective functionobjective function, n.: The score that an optimization process tries to push as low, or as high, as possible. For an AI, it defines what 'doing well' means during training.