empirical risk minimization

The principle of choosing, from a class of candidate models (a hypothesis spacehypothesis space, n.: The full menu of possible models a learning method is allowed to choose from — for example, all straight lines. Learning means picking the item on the menu that best fits the data.), the one that minimizes the average lossloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down. on the training data — the empirical risk — as a proxy for the expected (true) risk over the unknown data distribution. Most supervised training, from ordinary least squaresordinary least squares, n.: The standard way to fit a line through data: choose the line that makes the total of the squared vertical distances from the points to the line as small as possible. and logistic regressionlogistic regression, n.: A simple, widely used method for yes-or-no predictions that outputs a probability — say, a 70% chance a customer cancels — by passing a weighted sum of inputs through an S-shaped curve. to neural networks trained by stochastic gradient descentstochastic gradient descent, n.: A faster version of gradient descent that estimates which way is downhill from a small random sample of the training data at each step, instead of checking the whole dataset., is ERM with a particular loss. Statistical learning theory asks when the empirical minimizer also nearly minimizes the true risk: uniform convergence bounds based on the VC dimensionVC dimension, n.: A measure of how flexible a family of prediction rules is: the largest number of points it can label in every possible way. More flexible families need more data to learn reliably. or related complexity measures guarantee it when the class is not too rich for the sample size, and the difference between the two risks is the generalization gapgeneralization gap, n.: The difference between how well a model does on the examples it trained on and how well it does on new ones. A big gap means it memorized rather than learned.. Because minimizing 0–1 classification error directly is computationally hard, convex surrogates such as hinge losshinge loss, n.: A way of scoring a classifier's mistakes that charges nothing for answers that are right with room to spare, and charges more and more the further an answer falls on the wrong side. and log losslog loss, n.: A score for probability predictions that punishes confident mistakes severely: saying '99% sure' and being wrong costs far more than saying '60% sure' and being wrong. Lower is better. are minimized instead. Structural risk minimization and regularizationregularization, n.: Any technique that stops a model from memorizing its training examples too closely — often by penalizing complexity — so that it works better on new data. add a complexity penalty to counter overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.; distributionally robust variants target distribution shiftdistribution shift, n.: When the data an AI system meets in real use differs from the data it was trained on — new users, places, times or habits — often making it perform worse than tests suggested..

Under plain ERM the model fits the majority group well and ignores the rare subgroup, so we switched to a group-robust objective.