model selection
In plain wordsChoosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best.
The task of choosing among candidate models (different algorithms, feature sets, complexities or hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data. settings) the one expected to generalize best. Fit to the training data is a misleading criterion, because more flexible models always fit better (overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.), so model selection either estimates out-of-sample performance or trades fit against complexity. Empirical approaches score candidates on a validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one. or by cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn.. Analytic approaches use penalized criteria such as the Akaike information criterionAkaike information criterion, n.: A score for comparing statistical models that rewards fitting the data well but charges a penalty for every extra adjustable number, to avoid picking needlessly complicated models. Lower is better., the Bayesian information criterionBayesian information criterion, n.: A score for comparing statistical models that, like AIC, rewards good fit and penalizes extra adjustable numbers, but penalizes them more as the dataset grows, favoring simpler models. Lower is better. and minimum description lengthminimum description length, n.: The principle that the best explanation of some data is the one that lets you write the data down most briefly, counting both the rule and its exceptions. Good theories compress., or the Bayesian model evidence (marginal likelihood), which automatically penalizes unneeded flexibility; structural risk minimization chooses complexity using generalization bounds. Because selection itself uses data, the chosen model's validation score is optimistically biased, so an independent test settest set, n.: A set of examples kept completely apart from training and used at the end to measure how well a model works on data it has never seen, like a sealed final exam. or nested cross-validation is needed for an honest estimate. In practice selection also weighs cost, latency, interpretability and robustness, and for very large models it often reduces to comparing checkpoints and a few architectures guided by scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments..
Model selection by 10-fold cross-validation favored depth-6 trees over both shallower and deeper ones.
Origin
A classical problem of statistics, formalized in the 1960s and 1970s by Colin Mallows's Cp statistic, Hirotugu Akaike's information criterion (1973), cross-validatory choice (Mervyn Stone, 1974) and Gideon Schwarz's Bayesian criterion (1978). Vladimir Vapnik and Alexey Chervonenkis's structural risk minimization (1970s) offered a learning-theory route.
Related
- cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn.
- validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one.
- Akaike information criterionAkaike information criterion, n.: A score for comparing statistical models that rewards fitting the data well but charges a penalty for every extra adjustable number, to avoid picking needlessly complicated models. Lower is better.
- Bayesian information criterionBayesian information criterion, n.: A score for comparing statistical models that, like AIC, rewards good fit and penalizes extra adjustable numbers, but penalizes them more as the dataset grows, favoring simpler models. Lower is better.
- overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding.
- hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking.
- minimum description lengthminimum description length, n.: The principle that the best explanation of some data is the one that lets you write the data down most briefly, counting both the rule and its exceptions. Good theories compress.
- bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between.
Sources
- Burnham, K. P. & Anderson, D. R. (2002). Model Selection and Multimodel Inference: A Practical Information-Theoretic Approach, 2nd ed. New York: Springer.
- Stone, M. (1974). Cross-Validatory Choice and Assessment of Statistical Predictions. Journal of the Royal Statistical Society, Series B 36(2).
Mentioned in
training settraining set, n.: The examples a machine-learning model learns from, like the worked problems a student practices on, kept separate from the examples later used to check how well it learned., AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed., bias–variance tradeoffbias–variance tradeoff, n.: The balancing act between models too simple to capture a pattern and models so flexible that they also learn the noise. The best predictions usually come from somewhere in between., cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn., hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking., k-means clusteringk-means clustering, n.: A simple way to split data into a chosen number of groups: guess group centers, assign every point to its nearest center, move each center to the middle of its points, and repeat., Occam's razorOccam's razor, n.: A rule of thumb for choosing between explanations: if two fit the evidence equally well, bet on the one that assumes less. In AI it motivates preferring simpler models, which are less likely to fit noise., silhouette scoresilhouette score, n.: A number between −1 and 1 that grades a grouping of data by checking, for each point, whether it sits closer to its own group than to the next-nearest group., underfittingunderfitting, n.: When a model is too simple, or trained too little, to capture the real pattern in the data, so it does poorly even on the examples it learned from., validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one., Akaike information criterionAkaike information criterion, n.: A score for comparing statistical models that rewards fitting the data well but charges a penalty for every extra adjustable number, to avoid picking needlessly complicated models. Lower is better., Alexey IvakhnenkoAlexey Ivakhnenko, prop. n.: A Soviet Ukrainian scientist who from 1965 built some of the first deep, many-layered learning networks, growing them layer by layer from data, decades before modern deep learning., Bayesian information criterionBayesian information criterion, n.: A score for comparing statistical models that, like AIC, rewards good fit and penalizes extra adjustable numbers, but penalizes them more as the dataset grows, favoring simpler models. Lower is better., minimum description lengthminimum description length, n.: The principle that the best explanation of some data is the one that lets you write the data down most briefly, counting both the rule and its exceptions. Good theories compress.