Akaike information criterion
In plain wordsA score for comparing statistical models that rewards fitting the data well but charges a penalty for every extra adjustable number, to avoid picking needlessly complicated models. Lower is better.
A criterion for model selectionmodel selection, n.: Choosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best. that estimates a model's expected out-of-sample predictive accuracy, measured by the Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right. from the true data-generating process, up to a constant. For a model with k estimated parameters and maximized likelihoodlikelihood, n.: A score for a possible explanation of some data: how probable that explanation, or that setting of a model, would have made the data actually seen. L̂, AIC = 2k − 2 ln L̂. Among candidates fitted to the same data, the one with the lowest AIC is preferred, and AIC differences can be converted into relative weights for model averaging. The 2k term corrects the optimism of the in-sample log-likelihood; a small-sample version, AICc, adds a further penalty when the number of observations is not much larger than k. AIC is asymptotically equivalent to leave-one-out cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn. for many models and targets prediction rather than identification of a 'true' model; unlike the Bayesian information criterionBayesian information criterion, n.: A score for comparing statistical models that, like AIC, rewards good fit and penalizes extra adjustable numbers, but penalizes them more as the dataset grows, favoring simpler models. Lower is better. it is not consistent and may keep favoring overly complex models as data grow. It needs likelihood-based models with well-defined parameter counts, which limits its use for neural networks.
AIC preferred ARIMA(2,1,1) over ARIMA(1,1,1) by 14 points.
Origin
Introduced by the Japanese statistician Hirotugu Akaike, who presented it at a 1971 symposium (published 1973) and set it out in 'A New Look at the Statistical Model Identification' (IEEE Transactions on Automatic Control, 1974). Akaike's abbreviation for 'an information criterion' came to be read as his own name. Clifford Hurvich and Chih-Ling Tsai popularized AICc in 1989.
Related
- Bayesian information criterionBayesian information criterion, n.: A score for comparing statistical models that, like AIC, rewards good fit and penalizes extra adjustable numbers, but penalizes them more as the dataset grows, favoring simpler models. Lower is better.
- model selectionmodel selection, n.: Choosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best.
- likelihoodlikelihood, n.: A score for a possible explanation of some data: how probable that explanation, or that setting of a model, would have made the data actually seen.
- Kullback–Leibler divergenceKullback–Leibler divergence, n.: A score for the cost of believing the wrong odds: on average, how much more surprised you will be by real outcomes than someone who knew the true probabilities. It is zero only when your odds are exactly right.
- cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn.
- ARIMAARIMA, n.: A classic statistical recipe for forecasting a series of numbers over time: remove any steady trend, then predict the next value from recent values and recent prediction errors.
- Bayesian information criterionBayesian information criterion, n.: A score for comparing statistical models that, like AIC, rewards good fit and penalizes extra adjustable numbers, but penalizes them more as the dataset grows, favoring simpler models. Lower is better.
Sources
- Akaike, H. (1973). Information Theory and an Extension of the Maximum Likelihood Principle. In 2nd International Symposium on Information Theory. Budapest: Akadémiai Kiadó.
- Akaike, H. (1974). A New Look at the Statistical Model Identification. IEEE Transactions on Automatic Control 19(6).
Mentioned in
ARIMAARIMA, n.: A classic statistical recipe for forecasting a series of numbers over time: remove any steady trend, then predict the next value from recent values and recent prediction errors., model selectionmodel selection, n.: Choosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best., Bayesian information criterionBayesian information criterion, n.: A score for comparing statistical models that, like AIC, rewards good fit and penalizes extra adjustable numbers, but penalizes them more as the dataset grows, favoring simpler models. Lower is better.