random search
In plain wordsTuning a model by trying settings picked at random from sensible ranges and keeping the best; surprisingly, this usually beats checking a neat grid of values.
A hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking. strategy that samples configurations independently at random from a specified distribution for each hyperparameter (for example, a log-uniform distribution for the learning rate) and keeps the best according to validation performance. For the same number of trials it usually finds better settings than grid searchgrid search, n.: Tuning a model by trying every combination from a short list of candidate settings, like testing every pairing on a menu, and keeping whichever combination scores best., because performance typically hinges on a small, problem-dependent subset of the hyperparameters: a grid of n points over d hyperparameters tries only about the d-th root of n distinct values of each, whereas n random draws can try up to n distinct values of every one. Random search is easily parallelized, can be stopped or extended at any time, and handles conditional and mixed-type search spaces without difficulty. It is the standard baseline for more sophisticated methods: Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next. uses past results to choose promising points, and multi-fidelity methods such as HyperbandHyperband, n.: A tuning method that starts many candidate settings on small budgets, keeps only the most promising ones, and gives the survivors more training time, so effort isn't wasted on losers. add early stopping of poor configurations to random sampling. More broadly, 'random search' also names a family of derivative-free optimization methods studied since the 1950s and 1960s.
Sixty trials of random search found a better configuration than our 500-point grid.
Origin
Random search as a derivative-free optimizer dates to the late 1950s and 1960s, with early analyses by Leonard Rastrigin (1963). Its use for hyperparameters became standard after James Bergstra and Yoshua Bengio's 2012 Journal of Machine Learning Research paper showed, empirically and theoretically, that it beats grid search when few hyperparameters matter.
Related
- grid searchgrid search, n.: Tuning a model by trying every combination from a short list of candidate settings, like testing every pairing on a menu, and keeping whichever combination scores best.
- hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking.
- Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next.
- HyperbandHyperband, n.: A tuning method that starts many candidate settings on small budgets, keeps only the most promising ones, and gives the survivors more training time, so effort isn't wasted on losers.
- hyperparameterhyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data.
- AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed.
- grid searchgrid search, n.: Tuning a model by trying every combination from a short list of candidate settings, like testing every pairing on a menu, and keeping whichever combination scores best.
Sources
- Bergstra, J. & Bengio, Y. (2012). Random Search for Hyper-Parameter Optimization. Journal of Machine Learning Research 13: 281–305.
Mentioned in
Bayesian optimizationBayesian optimization, n.: A smart way to tune settings when each test is expensive: build a statistical guess of how settings affect results, then try whichever setting that guess says is most promising next., grid searchgrid search, n.: Tuning a model by trying every combination from a short list of candidate settings, like testing every pairing on a menu, and keeping whichever combination scores best., hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking., HyperbandHyperband, n.: A tuning method that starts many candidate settings on small budgets, keeps only the most promising ones, and gives the survivors more training time, so effort isn't wasted on losers.