ensemble learning
In plain wordsCombining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model.
The practice of training multiple models (base learners) and combining their predictions — by majority or weighted voting, averaging of scores or probabilities, or a learned combiner — so that the combination is more accurate, more robust or better at estimating uncertainty than its members individually. Ensembles work when members are individually competent and make diverse, partly uncorrelated errors. Diversity is induced by resampling the data (baggingbagging, n.: Training many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error.), randomizing features or splits (random forestsrandom forest, n.: A prediction method that builds hundreds of decision trees, each from a random sample of the data and features, and lets them vote. The crowd is far more reliable than any single tree.), training sequentially on errors (boostingboosting, n.: Building a strong predictor from many weak ones trained one after another, where each new model concentrates on the mistakes the previous ones made.), combining different algorithms (stackingstacking, n.: Combining several different models by training one more model to decide how much to trust each of them, based on how they performed on data they had not seen.) or, for neural networks, varying random initializations (deep ensembles). Averaging mainly reduces variance; boosting mainly reduces bias. Disagreement among members is a common practical estimate of predictive uncertainty. The costs are extra computation and memory and a loss of interpretability; knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run. can compress an ensemble into one model.
Averaging five seeds of the same network gave a cheap ensemble and better-calibrated probabilities.
Origin
Combining independent judgments has old roots, such as Condorcet's 1785 jury theorem. In machine learning, Lars Hansen and Peter Salamon's 'Neural Network Ensembles' (1990), David Wolpert's stacking (1992), Leo Breiman's bagging (1994–1996) and the boosting work of Robert Schapire and Yoav Freund (1990–1997) established the field. Large blends won the Netflix Prize in 2009.
Usage note
Not to be confused with mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network., which routes each input to a few specialized sub-networks inside one model, or with model mergingmodel merging, n.: Combining two or more trained AI models into one by blending their internal numbers directly, so the result shares their skills without any further training., which averages weights rather than predictions.
Related
- baggingbagging, n.: Training many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error.
- boostingboosting, n.: Building a strong predictor from many weak ones trained one after another, where each new model concentrates on the mistakes the previous ones made.
- random forestrandom forest, n.: A prediction method that builds hundreds of decision trees, each from a random sample of the data and features, and lets them vote. The crowd is far more reliable than any single tree.
- stackingstacking, n.: Combining several different models by training one more model to decide how much to trust each of them, based on how they performed on data they had not seen.
- knowledge distillationknowledge distillation, n.: Training a smaller or cheaper 'student' AI to copy the behavior of a larger 'teacher' AI, so the student keeps much of the teacher's skill while being faster to run.
- Netflix PrizeNetflix Prize, prop. n.: A 2006–2009 contest in which Netflix offered a million dollars to anyone who could predict customers' movie ratings 10 percent better than its own system. It popularized machine-learning competitions and model blending.
- mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
- model mergingmodel merging, n.: Combining two or more trained AI models into one by blending their internal numbers directly, so the result shares their skills without any further training.
Sources
- Hansen, L. K. & Salamon, P. (1990). Neural Network Ensembles. IEEE Transactions on Pattern Analysis and Machine Intelligence, 12(10), 993–1001.
- Dietterich, T. G. (2000). Ensemble Methods in Machine Learning. In Multiple Classifier Systems (LNCS 1857), 1–15. Springer.
Mentioned in
overfittingoverfitting, n.: When a model learns its training examples too well — quirks and noise included — so it does great on data it has seen but poorly on anything new, like a student who memorizes answers without understanding., AdaBoostAdaBoost, prop. n.: One of the first practical boosting methods: it trains simple models in turn, giving extra weight to examples the earlier models got wrong, then lets all the models vote, weighted by accuracy., baggingbagging, n.: Training many copies of a model, each on a different random resample of the data, and averaging their predictions. The averaging cancels out much of each copy's random error., boostingboosting, n.: Building a strong predictor from many weak ones trained one after another, where each new model concentrates on the mistakes the previous ones made., dropoutdropout, n.: A training trick that randomly switches off a fraction of a network's neurons at each step, so the network cannot lean too heavily on any one of them and learns sturdier patterns., isolation forestisolation forest, n.: A method for spotting unusual data points by repeatedly slicing the data at random: odd points get cut off from the rest after only a few slices, while ordinary points take many., Leo BreimanLeo Breiman, prop. n.: An American statistician who invented random forests and bagging, decision-tree methods still in wide use, and urged statisticians to judge models by how well they predict., mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network., model mergingmodel merging, n.: Combining two or more trained AI models into one by blending their internal numbers directly, so the result shares their skills without any further training., random forestrandom forest, n.: A prediction method that builds hundreds of decision trees, each from a random sample of the data and features, and lets them vote. The crowd is far more reliable than any single tree., Shapley valueShapley value, n.: A rule from game theory for splitting a shared reward fairly among team members, by averaging how much each member adds when joining the team in every possible order., XGBoostXGBoost, prop. n.: A fast, free software tool that makes predictions by adding up hundreds of small decision trees, each one fixing the mistakes of those before it. For years it was behind many winning entries in competitions on spreadsheet-style data., Bayesian neural networkBayesian neural network, n.: A neural network that keeps a whole range of plausible settings for its internal numbers instead of one best guess, so that it can say how unsure it is about each prediction., mode connectivitymode connectivity, n.: The discovery that two separately trained versions of a network, though their settings look very different, can usually be joined by a path along which the model keeps working well., model cascademodel cascade, n.: A chain of AI models ordered from cheapest to most capable: each request goes to the cheap one first and moves up the chain only if the answer does not look good enough., model soupmodel soup, n.: A single model made by averaging the internal numbers of several copies of a model that were fine-tuned in slightly different ways, often doing better than any one of them., Netflix PrizeNetflix Prize, prop. n.: A 2006–2009 contest in which Netflix offered a million dollars to anyone who could predict customers' movie ratings 10 percent better than its own system. It popularized machine-learning competitions and model blending., stackingstacking, n.: Combining several different models by training one more model to decide how much to trust each of them, based on how they performed on data they had not seen.