associative memory
In plain wordsA memory that is searched by content rather than by location: given part of a stored pattern, or a noisy copy of it, it recalls the whole, much as a few notes can bring back an entire song.
A system that stores patterns so that a cue, either a fragment or corrupted version of a stored pattern (auto-association) or an associated key (hetero-association), retrieves the stored item without any address. In neural-network models, storage is typically Hebbian: a correlation-matrix memory sums outer products of key and value vectors, and recall multiplies the cue by that matrix; in the Hopfield networkHopfield network, n.: A network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint., stored patterns are minima of an energy function and recall is descent to the nearest minimum. Classical capacity is small, roughly 0.14 patterns per neuron for the Hopfield network before recall breaks down, but modern Hopfield networks with sharper energy functions can store exponentially many patterns, and their one-step update has the same form as transformer attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones.. The idea also frames transformer feed-forward layers as key–value memories and motivates memory-augmented networks and sequence models whose memory is updated at test time.
Viewed as an associative memory, the attention layer retrieves the stored value whose key best matches the query.
Origin
Early neural models include Karl Steinbuch's Lernmatrix (1961) and the correlation-matrix memories of Teuvo Kohonen and of James Anderson (both 1972); John Hopfield's 1982 network gave the energy-based formulation. Dmitry Krotov and Hopfield's dense associative memories (2016) and Hubert Ramsauer and colleagues' 'Hopfield Networks is All You Need' (2020) connected the idea to attention.
Usage note
In hardware, content-addressable memory (CAM) means circuits that compare a search word with all stored words in parallel, as in network routers. In psychology, associative memory refers to learned links between items, the tradition from which the computational idea grew.
Related
- Hopfield networkHopfield network, n.: A network of on–off units, all connected to one another, that stores patterns as stable states. Given a partial or noisy pattern, it settles into the closest stored one, like recalling a memory from a hint.
- Hebbian learningHebbian learning, n.: The idea that a connection between two brain cells grows stronger when they are active at the same time — the principle behind the slogan 'cells that fire together wire together'.
- attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones.
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- energy-based modelenergy-based model, n.: A model that scores how well things fit together by giving each possibility an 'energy': low for plausible combinations, high for implausible ones, like a ball settling into the lowest valleys.
- Boltzmann machineBoltzmann machine, n.: A network of randomly flipping on–off units, including hidden ones, that learns the statistical patterns in its training data and can then produce new examples like them.
Sources
- Kohonen, T. (1972). Correlation Matrix Memories. IEEE Transactions on Computers C-21(4), 353–359.
- Hopfield, J. J. (1982). Neural Networks and Physical Systems with Emergent Collective Computational Abilities. PNAS 79(8), 2554–2558.
- Ramsauer, H. et al. (2021). Hopfield Networks is All You Need. ICLR 2021.
Mentioned in
Shun-ichi AmariShun-ichi Amari, prop. n.: A Japanese mathematician who trained multilayer neural networks by gradient descent as early as 1967 and founded information geometry, the source of the 'natural gradient' method., Teuvo KohonenTeuvo Kohonen, prop. n.: A Finnish engineer who invented the self-organizing map, a neural network that arranges complex data on a simple grid so that similar items end up next to each other.