hyperparameter tuning
In plain wordsTrying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking.
The search for the hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data. values that give a learning algorithm its best performance, measured on held-out data through a validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one. or cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn.. It is an expensive black-box optimization problem: each evaluation requires training a model, the objective is noisy and has no usable gradient, and the search space mixes continuous, integer, categorical and conditional variables. Strategies range from manual tuning guided by experience, through exhaustive grid searchgrid search, n.: Tuning a model by trying every combination from a short list of candidate settings, like testing every pairing on a menu, and keeping whichever combination scores best. and random searchrandom search, n.: Tuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values., to model-based Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next., evolutionary and population-based methods, and multi-fidelity schemes such as successive halving and HyperbandHyperband, n.: A tuning method that starts many candidate settings on small budgets, keeps only the most promising ones, and gives the survivors more training time, so effort isn't wasted on losers., which use small budgets (fewer epochs or less data) to stop unpromising configurations early. For very large models, tuning is done at small scale and transferred with scaling rules. Tuning is a core component of AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed., and open-source tools such as Optuna and Ray Tune, as well as experiment-tracking platforms, automate it.
The sweep ran 200 trials overnight; learning rate and weight decay mattered, and nothing else did.
Origin
Hand tuning is as old as machine learning. Automated tuning matured in the early 2010s: James Bergstra and colleagues' tree-structured Parzen estimator (2011), Bergstra and Yoshua Bengio's random-search study (2012) and the Spearmint Bayesian-optimization tool of Jasper Snoek, Hugo Larochelle and Ryan Adams (2012) made hyperparameter optimization a research field of its own.
Usage note
Reported gains should state the tuning budget: tuning a new method more thoroughly than its baselinesbaseline, n.: A simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value., or tuning on the test set, produces misleading comparisons.
Related
- hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data.
- grid searchgrid search, n.: Tuning a model by trying every combination from a short list of candidate settings, like testing every pairing on a menu, and keeping whichever combination scores best.
- random searchrandom search, n.: Tuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values.
- Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next.
- HyperbandHyperband, n.: A tuning method that starts many candidate settings on small budgets, keeps only the most promising ones, and gives the survivors more training time, so effort isn't wasted on losers.
- AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed.
- cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn.
- model selectionmodel selection, n.: Choosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best.
Sources
- Bergstra, J., Bardenet, R., Bengio, Y. & Kégl, B. (2011). Algorithms for Hyper-Parameter Optimization. Advances in Neural Information Processing Systems 24.
- Snoek, J., Larochelle, H. & Adams, R. P. (2012). Practical Bayesian Optimization of Machine Learning Algorithms. Advances in Neural Information Processing Systems 25.
- Feurer, M. & Hutter, F. (2019). Hyperparameter Optimization. In Hutter, F., Kotthoff, L. & Vanschoren, J. (eds.), Automated Machine Learning: Methods, Systems, Challenges. Springer.
Mentioned in
optimizationoptimization, n.: Finding the best possible choice according to some score, such as the lowest error or the highest reward. Training an AI is optimization: adjusting its settings until its mistakes are as small as possible., training settraining set, n.: The examples a machine-learning model learns from, like the worked problems a student practices on, kept separate from the examples later used to check how well it learned., AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed., baselinebaseline, n.: A simple reference method that a new model must beat to show it is worth anything, such as always guessing the most common answer or reusing yesterday's value., Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next., cross-validationcross-validation, n.: A fair way to test a model when data are limited: split the data into several parts, train on all but one, test on the part left out, and rotate so every part takes a turn., data leakagedata leakage, n.: A mistake in building a model that lets it peek at clues it will not have in real use, such as hints about the very answer it must predict, so it shines in testing and then disappoints., experiment trackingexperiment tracking, n.: Keeping an automatic logbook of every model-training attempt — settings, data, code and results — so a team can compare attempts, repeat the good ones and see what changed., grid searchgrid search, n.: Tuning a model by trying every combination from a short list of candidate settings, like testing every pairing on a menu, and keeping whichever combination scores best., hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data., model selectionmodel selection, n.: Choosing, from several candidate models or settings, the one most likely to work best on new data, rather than the one that merely fits the existing data best., multi-armed banditmulti-armed bandit, n.: A puzzle named after a gambler choosing between slot machines: you must keep picking among options whose payoffs are unknown, learning which is best while wasting as few tries as possible on the others., random searchrandom search, n.: Tuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values., validation setvalidation set, n.: A set of examples kept out of training and used during development to compare model versions and settings, like a practice exam taken before the real one., AlgoPerfAlgoPerf, prop. n.: A public contest that measures which methods for training neural networks reach a set level of accuracy fastest on identical hardware, across several kinds of tasks, without being hand-tuned for each one., HyperbandHyperband, n.: A tuning method that starts many candidate settings on small budgets, keeps only the most promising ones, and gives the survivors more training time, so effort isn't wasted on losers., maximal update parametrizationmaximal update parametrization, n.: A recipe for scaling a neural network's starting values and learning speeds with its size so that settings tuned on a small, cheap model work unchanged on a huge one., neural architecture searchneural architecture search, n.: Using an automated search, rather than human trial and error, to design the structure of a neural network: which layers to use, how many, and how to connect them., RayRay, prop. n.: Free software that spreads Python programs, including AI training and model serving, across many computers and graphics chips so that big jobs run in parallel., tabular foundation modeltabular foundation model, n.: A neural network pretrained on millions of made-up tables so that, given a new spreadsheet of labeled examples, it can predict the missing answers at once, without being retrained for that table., Weights & BiasesWeights & Biases, prop. n.: A company and online service that machine-learning teams use to log, chart and compare their training experiments, models and AI-app test results in shared dashboards., YOLO runYOLO run, n.: Slang for launching a big, expensive training run without the usual careful small-scale tests first — changing many things at once and trusting experience and instinct that it will work. YOLO means 'you only live once'.