Skip to content
Last Dictionary
Plane
Language Models & TransformersDeep LearningMathematics & Statistics

state space model

nounAlso: SSM, structured state space model, structured state space sequence model, S4, HiPPO, Hyena

In plain wordsA type of sequence model that reads text like a running tally, updating a fixed-size internal summary at each step, so its memory and cost do not balloon as texts get longer.

  1. 1

    Deep learning

    A sequence-modeling layer derived from linear state-space systems in control theory, in which a hidden state evolves under a linear rule driven by the input and is read out linearly, discretized for sequences of tokens. Because early designs are linear and time-invariant, the output over a whole sequence can be computed in parallel as one long convolution during training and step by step as a recurrence with constant memory during generation. S4 (2021) made this practical with a structured parameterization of the state matrix rooted in HiPPO, a theory of compressing a signal's history online, and excelled on long-range benchmarks. Language modeling required input-dependent ('selective') parameters, introduced in MambaMamba, prop. n.: A 2023 alternative to the transformer design that processes text with a continually updated summary which decides what to keep or forget, letting it handle long sequences quickly. (2023), which narrowed the gap to transformerstransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.. SSMs remain weaker at exact copying and recall from long context, since their fixed state must compress the past, which motivates hybridshybrid architecture, n.: An AI model design that mixes two kinds of building blocks — usually a few standard attention layers among cheaper, memory-saving layers — to keep most of the quality at lower cost on long inputs. with attention.

    The SSM layers keep a 16-dimensional state per channel, so generation memory is constant.

  2. 2

    Statistics and control

    A model of a dynamic system as an unobserved state that evolves over time, together with noisy observations of that state, as in the Kalman filter; standard in time-series analysis, econometrics and control engineering. The deep-learning sense borrows its linear form.

ht=Aˉ ht−1+Bˉ xt,yt=C hth_t = \bar{A}\, h_{t-1} + \bar{B}\, x_t, \qquad y_t = C\, h_t

Origin

State-space methods entered control engineering with Rudolf Kálmán around 1960. At Stanford, Albert Gu and colleagues introduced HiPPO (2020) and S4 (Albert Gu, Karan Goel and Christopher Ré, October 2021), establishing deep SSMs; Mamba (Albert Gu and Tri Dao, December 2023) adapted them to language.

Related

See also
  • MambaMamba, prop. n.: A 2023 alternative to the transformer design that processes text with a continually updated summary which decides what to keep or forget, letting it handle long sequences quickly.
  • linear attentionlinear attention, n.: A family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words.
  • hybrid architecturehybrid architecture, n.: An AI model design that mixes two kinds of building blocks — usually a few standard attention layers among cheaper, memory-saving layers — to keep most of the quality at lower cost on long inputs.
  • recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood.
  • RWKVRWKV, prop. n.: An open-source language model design that works like an old-style recurrent network when writing text, using constant memory, yet can be trained quickly in parallel like a transformer.
  • long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.
Contrast with
  • transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.

Sources

  1. Gu, A., Goel, K. & Ré, C. (2022). Efficiently Modeling Long Sequences with Structured State Spaces. ICLR 2022. arxiv.org/abs/2111.00396

Mentioned in

transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it., architecturearchitecture, n.: The overall design of an AI model, meaning what kinds of parts it has and how they are connected, as opposed to the specific numbers it learns during training., big O notationbig O notation, n.: Shorthand for how the cost of a computation grows as the problem gets bigger. Saying attention costs 'O(n²)' means doubling the length of the text roughly quadruples the work., long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible., recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood., statestate, n.: A snapshot of the situation an AI agent is in — for a chess program, the arrangement of pieces and whose turn it is — containing what it needs to decide what to do next., hybrid architecturehybrid architecture, n.: An AI model design that mixes two kinds of building blocks — usually a few standard attention layers among cheaper, memory-saving layers — to keep most of the quality at lower cost on long inputs., linear attentionlinear attention, n.: A family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words., liquid neural networkliquid neural network, n.: A small, brain-inspired network for time-varying data whose neurons change how quickly they react depending on their input, letting a few neurons handle tasks such as steering a car., MambaMamba, prop. n.: A 2023 alternative to the transformer design that processes text with a continually updated summary which decides what to keep or forget, letting it handle long sequences quickly., nested learningnested learning, n.: A Google research idea that treats an AI model as layers of learning processes updating at different speeds, with the aim of letting models keep learning without forgetting old skills., RWKVRWKV, prop. n.: An open-source language model design that works like an old-style recurrent network when writing text, using constant memory, yet can be trained quickly in parallel like a transformer., test-time trainingtest-time training, n.: Letting an AI model briefly update itself on each new problem it is given, learning from that problem's own clues before answering, instead of keeping its knowledge frozen after training.