hypernetwork

A neural networkneural network, n.: A kind of computer program that learns a task from examples by tuning thousands to billions of connections between simple number-crunching units, an arrangement loosely inspired by nerve cells in the brain. that outputs all or part of the weightsweights, n.: The learned numbers inside a neural network that set how strongly each part influences the next. The finished set of weights is, in effect, the trained model itself. of another network, the main or target network, which then processes the actual input; both are trained end to end by backpropagating the main network's loss into the hypernetwork's parameters. Conditioning the hypernetwork on a layer index, task description, input or time step lets it generate different weights for different layers, tasks or inputs, a relaxed form of weight sharingweight sharing, n.: Reusing the same learned numbers in many places in a network, for example one edge detector applied at every spot in an image, so the network needs fewer parameters and learns from less data. and a compact parameterization. Applications include multi-task and continual learningcontinual learning, n.: The ability of an AI to keep learning new things over time from a stream of experience, without forgetting what it already knew and without being retrained from scratch., neural architecture searchneural architecture search, n.: Using an automated search, rather than human trial and error, to design the structure of a neural network: which layers to use, how many, and how to connect them., meta-learningmeta-learning, n.: Training a system across many different tasks so that it gets better at learning itself, letting it pick up a new task quickly from only a few examples., personalization, implicit neural representations and generating LoRAlow-rank adaptation, n.: A cheap way to customize a big AI model: instead of changing all its billions of numbers, you train a small add-on of a few million numbers that slightly adjusts how it behaves. adapters from text descriptions. Fast-weight programmers, in which one network rapidly rewrites another's weights, are a closely related older idea, and linear attentionlinear attention, n.: A family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words. can be read as such a fast-weight system.

A small hypernetwork generates a LoRA adapter for each new task from its description.