hidden layer
In plain wordsAny layer of a neural network between the input and the output. Its values are never shown to the user; they are the network's internal working-out.
A layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage. of a neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain. lying between the input and output layers, whose activations are not specified by the training data but are learned as internal representations (representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.). Given enough units, a network with one hidden layer and a suitable nonlinear (non-polynomial) activation functionactivation function, n.: The simple rule each artificial neuron applies to its total input to decide its output, such as 'keep positive values, turn negatives into zero'. Without it, a deep network would be no smarter than a single layer. can approximate any continuous function on a closed, bounded domain (universal approximation theoremuniversal approximation theorem, n.: A mathematical result showing that a neural network with enough neurons can imitate essentially any smooth input-to-output relationship as closely as you like, though it does not say how to find that network.); networks with many hidden layers are called deep. The number of hidden layers (depth) and of units per layer (width) are central hyperparametershyperparameter, n.: A setting chosen by people before training, such as how fast a model learns or how big it is, as opposed to the internal values the model learns by itself from data.. The vector of a hidden layer's outputs is its hidden state or hidden representation; in transformers the 'hidden size' (d_model) is the width of the residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document..
Doubling the hidden size improved validation loss more than adding two hidden layers.
Origin
The 'hidden unit' terminology was popularized in 1980s connectionism, notably by the Boltzmann machine work of David Ackley, Geoffrey Hinton and Terrence Sejnowski (1985), which divided units into visible and hidden, and by the backpropagation paper of David Rumelhart, Hinton and Ronald Williams (1986), which showed hidden units learning useful internal features.
Related
- layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage.
- multilayer perceptronmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model.
- universal approximation theoremuniversal approximation theorem, n.: A mathematical result showing that a neural network with enough neurons can imitate essentially any smooth input-to-output relationship as closely as you like, though it does not say how to find that network.
- representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand.
- residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.
- Boltzmann machineBoltzmann machine, n.: A network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them.
- latent spacelatent space, n.: The hidden internal map a model builds of its data, where each item becomes a point and similar items sit near each other. Moving smoothly through it gradually changes what the model represents or generates.
Mentioned in
layerlayer, n.: One stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage., David RumelhartDavid Rumelhart, prop. n.: An American psychologist who, with Geoffrey Hinton and Ronald Williams, showed in 1986 how to train multilayer neural networks, launching the modern revival of the field., multilayer perceptronmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model., representation learningrepresentation learning, n.: Letting a computer work out for itself how best to describe its data, such as the edges and shapes in images or the meanings of words, instead of relying on descriptions people design by hand., universal approximation theoremuniversal approximation theorem, n.: A mathematical result showing that a neural network with enough neurons can imitate essentially any smooth input-to-output relationship as closely as you like, though it does not say how to find that network., XOR problemXOR problem, n.: The classic puzzle of teaching a single artificial neuron the 'one or the other but not both' rule. It cannot be done with one straight dividing line, which showed why networks need hidden layers., radial basis function networkradial basis function network, n.: A three-layer network whose hidden neurons each respond to inputs near a particular reference point, like bells placed across the data; the output is a weighted mix of the bells.