neural architecture search
In plain wordsUsing an automated search, rather than human trial and error, to design the structure of a neural network: which layers to use, how many, and how to connect them.
The automated design of neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain. architectures; in the widely used taxonomy of Thomas Elsken, Jan Hendrik Metzen and Frank Hutter (2019), a method is defined by three choices: a search space (the operations and connection patterns allowed, often as repeated 'cells'), a search strategy (reinforcement learning, evolutionary algorithms, Bayesian optimization, random search or gradient-based relaxation) and a performance-estimation strategy (full training, short training, learning-curve extrapolation or shared-weight proxies). Early methods trained thousands of candidates and consumed thousands of GPU-days; weight sharingweight sharing, n.: Reusing the same learned numbers in many places in a network, for example one edge detector applied at every spot in an image, so the network needs fewer parameters and learns from less data. ('one-shot') methods such as ENAS and differentiable methods such as DARTS, which relax discrete choices into continuous weights optimized by gradient descentgradient descent, n.: A method for training models by repeatedly nudging every setting a small step in whichever direction most reduces the error, like a hiker who cannot see the valley but keeps stepping wherever the ground tilts down., cut this to a few GPU-days. NAS produced NASNet, AmoebaNet, MnasNet and the EfficientNet base network, and hardware-aware NAS optimizes for latency on target devices. Critics showed that well-tuned random search is a strong baseline and that reported gains depend heavily on the search space and training recipe. NAS is part of AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed..
Hardware-aware NAS found a model 1.8 times faster on the phone at the same accuracy.
Origin
Evolving network topologies (neuroevolution) dates to the late 1980s; NEAT (Kenneth Stanley and Risto Miikkulainen, 2002) became its best-known method. The term was popularized by Barret Zoph and Quoc Le's 'Neural Architecture Search with Reinforcement Learning' (arXiv, November 2016; ICLR 2017), in which a recurrent controller trained by reinforcement learning proposed architectures; DARTS (Hanxiao Liu, Karen Simonyan and Yiming Yang, 2018) made the search differentiable.
Related
- AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed.
- hyperparameter tuninghyperparameter tuning, n.: Trying out different settings for a model's training, such as learning speed or model size, and keeping the combination that performs best on data held back for checking.
- weight sharingweight sharing, n.: Reusing the same learned numbers in many places in a network, for example one edge detector applied at every spot in an image, so the network needs fewer parameters and learns from less data.
- evolutionary computationevolutionary computation, n.: Problem-solving by simulated evolution, much as breeders improve crops: keep many candidate answers, let the better ones produce slightly altered offspring, drop the worse, and repeat for many rounds.
- hypernetworkhypernetwork, n.: A neural network whose job is to produce the settings, or weights, of another neural network, so that one model can generate many specialized models on demand.
- depthwise separable convolutiondepthwise separable convolution, n.: A cheaper way to do the main operation of image networks: first scan each channel of the input separately, then mix the channels together with a tiny one-by-one step.
Sources
- Zoph, B. & Le, Q. V. (2016). Neural Architecture Search with Reinforcement Learning. arXiv:1611.01578; ICLR 2017. arxiv.org/abs/1611.01578
Mentioned in
architecturearchitecture, n.: The overall design of an AI model, meaning what kinds of parts it has and how they are connected, as opposed to the specific numbers it learns during training., AutoMLAutoML, n.: Software that automates the trial-and-error parts of building a machine-learning model, such as choosing the algorithm, its settings and the data-preparation steps, so less expert effort is needed., weight sharingweight sharing, n.: Reusing the same learned numbers in many places in a network, for example one edge detector applied at every spot in an image, so the network needs fewer parameters and learns from less data., depthwise separable convolutiondepthwise separable convolution, n.: A cheaper way to do the main operation of image networks: first scan each channel of the input separately, then mix the channels together with a tiny one-by-one step., EfficientNetEfficientNet, prop. n.: A family of image-recognition networks from 2019 that grows a small base design in a balanced way, deeper, wider and sharper together, to get more accuracy for each unit of computing., Gumbel-softmaxGumbel-softmax, n.: A mathematical trick that lets a neural network make a choice among separate options, such as picking one word or one category, while still learning through small, smooth adjustments., hypernetworkhypernetwork, n.: A neural network whose job is to produce the settings, or weights, of another neural network, so that one model can generate many specialized models on demand., MobileNetMobileNet, prop. n.: A family of compact image-recognition networks from Google, designed to run quickly on phones and other small devices with little loss of accuracy., neuroevolutionneuroevolution, n.: Training or designing neural networks by simulated evolution — breeding and mutating whole networks and keeping the best performers — instead of, or alongside, the usual gradient-based training., TinyMLTinyML, n.: Machine learning shrunk to run on tiny, cheap, battery-powered chips inside sensors, wearables or appliances, using so little power that a coin-sized battery can last for months or years.