Skip to content
Last Dictionary
Plane
Language Models & TransformersDeep Learning

linear attention

nounAlso: linear transformer, kernelized attention, Performer, RetNet, gated linear attention, DeltaNet, Gated DeltaNet

In plain wordsA family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words.

  1. A family of attention approximations or replacements whose cost grows linearly with sequence length, obtained by removing or replacing the softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.. If similarity is a product of feature maps φ(q)·φ(k), the associativity of matrix products lets the model combine keys and values first and never form the n × n score matrix; in the causal case, the running sum of value–key outer products acts as a fixed-size matrix-valued state updated once per token, making the model a recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood. with constant memory during generation. Plain linear attention underperforms softmax attention, especially at recall from long context, because a fixed state must compress the whole past. Later variants add decay and gates (RetNet, gated linear attention, MambaMamba, prop. n.: A 2023 alternative to the transformer design that processes text with a continually updated summary which decides what to keep or forget, letting it handle long sequences quickly.-2) or delta-rule updates that overwrite stale associations (DeltaNet, Gated DeltaNet), and are often mixed with some full-attention layers in hybrid modelshybrid architecture, n.: An AI model design that mixes two kinds of building blocks — usually a few standard attention layers among cheaper, memory-saving layers — to keep most of the quality at lower cost on long inputs..

    Linear attention keeps a 128 × 128 state per head, so memory stays flat no matter how long the chat runs.

outt=(∑j≤tvj ϕ(kj)⊤)ϕ(qt)ϕ(qt)⊤∑j≤tϕ(kj)\mathrm{out}_t = \frac{\left(\sum_{j \le t} v_j\, \phi(k_j)^{\top}\right) \phi(q_t)}{\phi(q_t)^{\top} \sum_{j \le t} \phi(k_j)}

Origin

Angelos Katharopoulos, Apoorv Vyas, Nikolaos Pappas and François Fleuret formalized linear attention and its equivalence to recurrent networks in 'Transformers are RNNs' (June 2020); Performer (2020) approximated softmax with random features. Imanol Schlag, Kazuki Irie and Jürgen Schmidhuber (2021) linked it to Schmidhuber's fast-weight programmers of the early 1990s.

Related

See also
  • state space modelstate space model, n.: A type of sequence model that reads text like a running tally, updating a fixed-size internal summary at each step, so its memory and cost do not balloon as texts get longer.
  • MambaMamba, prop. n.: A 2023 alternative to the transformer design that processes text with a continually updated summary which decides what to keep or forget, letting it handle long sequences quickly.
  • RWKVRWKV, prop. n.: An open-source language model design that works like an old-style recurrent network when writing text, using constant memory, yet can be trained quickly in parallel like a transformer.
  • hybrid architecturehybrid architecture, n.: An AI model design that mixes two kinds of building blocks — usually a few standard attention layers among cheaper, memory-saving layers — to keep most of the quality at lower cost on long inputs.
  • recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood.
  • sparse attentionsparse attention, n.: Attention in which each word looks at only a selected subset of the other words instead of all of them, so very long texts can be processed at a fraction of the cost.
  • long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.
Contrast with
  • scaled dot-product attentionscaled dot-product attention, n.: The standard formula for attention in transformers: score how well each question matches each label, scale the scores down, turn them into percentages, and use those to blend the stored information.

Sources

  1. Katharopoulos, A., Vyas, A., Pappas, N. & Fleuret, F. (2020). Transformers are RNNs: Fast Autoregressive Transformers with Linear Attention. ICML 2020. arxiv.org/abs/2006.16236
  2. Schlag, I., Irie, K. & Schmidhuber, J. (2021). Linear Transformers Are Secretly Fast Weight Programmers. ICML 2021.

Mentioned in

transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., big O notationbig O notation, n.: Shorthand for how the cost of a computation grows as the problem gets bigger. Saying attention costs 'O(n²)' means doubling the length of the text roughly quadruples the work., FlashAttentionFlashAttention, prop. n.: A faster way for AI chips to compute attention, the step in which a language model relates every word to every other. It gets exactly the same answer while moving far less data around the chip., long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible., hybrid architecturehybrid architecture, n.: An AI model design that mixes two kinds of building blocks — usually a few standard attention layers among cheaper, memory-saving layers — to keep most of the quality at lower cost on long inputs., hypernetworkhypernetwork, n.: A neural network whose job is to produce the settings, or weights, of another neural network, so that one model can generate many specialized models on demand., MambaMamba, prop. n.: A 2023 alternative to the transformer design that processes text with a continually updated summary which decides what to keep or forget, letting it handle long sequences quickly., RWKVRWKV, prop. n.: An open-source language model design that works like an old-style recurrent network when writing text, using constant memory, yet can be trained quickly in parallel like a transformer., sliding window attentionsliding window attention, n.: Attention in which each word looks only at a fixed number of nearby earlier words rather than the whole text, keeping the cost of very long inputs under control., sparse attentionsparse attention, n.: Attention in which each word looks at only a selected subset of the other words instead of all of them, so very long texts can be processed at a fraction of the cost., state space modelstate space model, n.: A type of sequence model that reads text like a running tally, updating a fixed-size internal summary at each step, so its memory and cost do not balloon as texts get longer., test-time trainingtest-time training, n.: Letting an AI model briefly update itself on each new problem it is given, learning from that problem's own clues before answering, instead of keeping its knowledge frozen after training.