layer
In plain wordsOne stage in a neural network: a group of artificial neurons that all take the same input, transform it, and pass the result on to the next stage.
A parameterized function that forms one stage of a neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain., mapping an input tensortensor, n.: A block of numbers with any number of dimensions: a single number, a list, a grid, or a stack of grids. AI software stores and processes all its data this way. to an output tensor; networks are built by composing layers, and a network's depth is the number of layers in sequence. Common types include fully connected (dense)fully connected layer, n.: A layer in which every input is connected to every output by its own adjustable weight. It is the most general and most common kind of neural-network layer., convolutionalconvolution, n.: An operation that slides a small grid of weights across an image or signal, multiplying and adding at each spot, to measure how strongly a pattern appears at every location., recurrentrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood., attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones., poolingpooling, n.: A step in an image network that shrinks its internal picture by summarizing each small patch with one number, usually the largest or the average, keeping what matters and discarding exact positions., normalizationnormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training., embeddingembedding, n.: A list of numbers that stands for a word, picture or other item, arranged so that similar items get similar lists. It lets a computer measure similarity of meaning as distance. and dropoutdropout, n.: A training trick that randomly switches off a fraction of a network's neurons at each step, so the network cannot lean too heavily on any one of them and learns sturdier patterns. layers; some, such as pooling, have no learnable parametersparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned.. In transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., 'layer' usually means a whole block (attention and feed-forwardfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge. sublayers wrapped in residual connectionsresidual connection, n.: A shortcut in a neural network that adds a block's input straight to its output, so each block only has to learn a correction. It is what makes networks hundreds of layers deep trainable. and normalization), so a '32-layer model' has 32 such blocks. Layers between input and output are hidden layershidden layer, n.: Any layer of a neural network between the input and the output. Its values are never shown to the user; they are the network's internal working-out.; early layers tend to compute simple, local features and later ones more abstract features.
The probe works best on the middle layers, around layer 16 of 32.
Origin
Layered networks go back to Frank Rosenblatt's perceptrons (1958), whose sensory, association and response units formed successive stages, and to hierarchical models of visual cortex such as Kunihiko Fukushima's neocognitron (1980). Deep-learning libraries such as Torch, Caffe, Keras and PyTorch later made the layer the basic unit of model code.
Related
- hidden layerhidden layer, n.: Any layer of a neural network between the input and the output. Its values are never shown to the user; they are the network's internal working-out.
- fully connected layerfully connected layer, n.: A layer in which every input is connected to every output by its own adjustable weight. It is the most general and most common kind of neural-network layer.
- neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain.
- deep learningdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract.
- residual streamresidual stream, n.: The running internal state that flows through a transformer model from layer to layer; each layer reads from it and adds its own contribution, like editors annotating a shared document.
- multilayer perceptronmultilayer perceptron, n.: A neural network made of several layers of artificial neurons, each connected to every neuron in the next layer. It is the simplest kind of deep network and a component of almost every modern model.
- parameterparameter, n.: One of the adjustable numbers inside an AI model that are tuned during training. Large language models have billions of them, and together they store what the model has learned.
Mentioned in
artificial neuronartificial neuron, n.: The basic building block of a neural network: a tiny calculator that multiplies each incoming number by a weight, adds them up, and passes the total through a simple function to decide its output., deep learningdeep learning, n.: Machine learning done with neural networks built from many stacked layers: raw data such as pixels, sound or text goes in at one end, and each layer turns what it receives into something more abstract., neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain., vectorvector, n.: An ordered list of numbers, like map coordinates. AI systems turn words, images and users into long lists of numbers so that similar things end up as nearby points., activationactivation, n.: The numbers a neural network computes inside itself as it processes an input — a reading of which of its artificial neurons are firing, and how strongly, at that moment., architecturearchitecture, n.: The overall design of an AI model, meaning what kinds of parts it has and how they are connected, as opposed to the specific numbers it learns during training., convolutionconvolution, n.: An operation that slides a small grid of weights across an image or signal, multiplying and adding at each spot, to measure how strongly a pattern appears at every location., feedforward neural networkfeedforward neural network, n.: A neural network laid out like an assembly line: each stage hands its result only to later stages, never back, so every answer depends on the current input alone and nothing is remembered between inputs., forward passforward pass, n.: The step where a neural network takes an input and runs it through its layers, one after another, to produce an output — its answer or prediction., fully connected layerfully connected layer, n.: A layer in which every input is connected to every output by its own adjustable weight. It is the most general and most common kind of neural-network layer., hidden layerhidden layer, n.: Any layer of a neural network between the input and the output. Its values are never shown to the user; they are the network's internal working-out., linear transformationlinear transformation, n.: A rule for moving every point in a space that keeps straight lines straight and the origin in place: rotating, stretching, reflecting or squashing everything evenly. Every grid of numbers describes one., matrixmatrix, n.: A rectangular grid of numbers arranged in rows and columns. Most of what a neural network has learned is stored in large grids like this., normalization layernormalization layer, n.: A layer that keeps the numbers passing through a neural network within a steady range by shifting and resizing them, so that deep networks learn faster and are less likely to break down during training., poolingpooling, n.: A step in an image network that shrinks its internal picture by summarizing each small patch with one number, usually the largest or the average, keeping what matters and discarding exact positions., activation spaceactivation space, n.: The imaginary space of all possible internal states of one layer of a neural network, where each input becomes a point and related meanings tend to share places or directions., early exitearly exit, n.: Letting an AI model stop partway through its layers and answer early when it is already confident, saving computing effort on easy inputs., gated linear unitgated linear unit, n.: A layer design in which one half of the computation acts as a set of dials, each between off and on, deciding how much of the other half passes through., transposed convolutiontransposed convolution, n.: An operation that roughly reverses an image network's shrinking steps, enlarging a small grid of features into a bigger one, as when a model paints a full-size image.