hybrid architecture
In plain wordsAn AI model design that mixes two kinds of building blocks — usually a few standard attention layers among cheaper, memory-saving layers — to keep most of the quality at lower cost on long inputs.
A model that interleaves different sequence-mixing layers, most often full self-attentionself-attention, n.: Attention in which a passage looks at itself: every word checks every other word in the same text to work out its meaning in context, such as which noun 'it' refers to. with fixed-state recurrent layers — state-spacestate space model, n.: A type of sequence model that reads text like a running tally, updating a fixed-size internal summary at each step, so its memory and cost do not balloon as texts get longer. layers such as MambaMamba, prop. n.: A 2023 alternative to the transformer design that processes text with a continually updated summary which decides what to keep or forget, letting it handle long sequences quickly., linear attentionlinear attention, n.: A family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words. variants or gated recurrences — and sometimes local attentionsliding window attention, n.: Attention in which each word looks only at a fixed number of nearby earlier words rather than the whole text, keeping the cost of very long inputs under control.. The recurrent layers do most of the sequence processing with constant memory per token, while a minority of attention layers, typically one in four to one in ten, supply the precise retrieval and in-context copying that fixed-state layers handle worst. The result is a much smaller KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch. and faster long-context generation at quality close to that of pure transformers. Examples include AI21's Jamba (2024), which combines Transformer, Mamba and MoEmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network. layers; and, from 2025, NVIDIA's Nemotron-H, MiniMax-01, IBM's Granite 4.0 (one attention layer per nine Mamba-2 layers) and Alibaba's Qwen3-Next (three Gated DeltaNet layers per attention layer). The term is also used loosely for any mix of architectural families, such as convolution–transformer hybrids in vision.
The hybrid keeps one attention layer in eight, so its KV cache is an eighth of the transformer baseline's.
Origin
Early attention–SSM hybrids include H3 (2022) and StripedHyena (2023). AI21 Labs' Jamba (March 2024), described by its makers as the first production-grade Mamba-based model, popularized the approach, and hybrids multiplied in 2025 as long-context and agentic workloads raised the cost of KV caches.
Related
- state space modelstate space model, n.: A type of sequence model that reads text like a running tally, updating a fixed-size internal summary at each step, so its memory and cost do not balloon as texts get longer.
- MambaMamba, prop. n.: A 2023 alternative to the transformer design that processes text with a continually updated summary which decides what to keep or forget, letting it handle long sequences quickly.
- linear attentionlinear attention, n.: A family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words.
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.
- long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible.
- mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
Sources
- Lieber, O., Lenz, B., Bata, H. et al. (2024). Jamba: A Hybrid Transformer-Mamba Language Model. arXiv.
Mentioned in
long contextlong context, n.: The ability of an AI model to work with very long inputs — whole books, codebases or hours of transcripts — in one go, and the techniques that make this possible., recurrent neural networkrecurrent neural network, n.: A neural network that reads a sequence, such as a sentence or an audio clip, one piece at a time, carrying a running summary forward so that what came earlier can shape how later pieces are understood., hybrid reasoning modelhybrid reasoning model, n.: A single AI model that can either answer right away or first think at length, with the user or developer choosing how much thinking to allow., linear attentionlinear attention, n.: A family of faster substitutes for attention whose cost grows in step with the length of the text rather than with its square, by keeping a running summary instead of comparing every pair of words., MambaMamba, prop. n.: A 2023 alternative to the transformer design that processes text with a continually updated summary which decides what to keep or forget, letting it handle long sequences quickly., RWKVRWKV, prop. n.: An open-source language model design that works like an old-style recurrent network when writing text, using constant memory, yet can be trained quickly in parallel like a transformer., state space modelstate space model, n.: A type of sequence model that reads text like a running tally, updating a fixed-size internal summary at each step, so its memory and cost do not balloon as texts get longer.