multi-head latent attention
In plain wordsAn attention design from the Chinese lab DeepSeek that compresses the stored information about earlier words into a much smaller summary, so long conversations need far less memory.
A variant of attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones. in which keys and values are reconstructed from a shared low-dimensional latent vector per token, so only that compressed latent, plus a small positional key, is stored in the KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.. Each token's hidden state is projected down to the latent; per-head keys and values are projected up from it, and these up-projections can be folded into the query and output projections so that decoding works directly on the cached latents. Because RoPErotary positional embedding, n.: A way of telling a transformer where each word is by rotating parts of its internal numbers by an angle that grows with position, so the model can read off how far apart two words are. would block this folding, position is carried in a separate small set of rotary dimensions shared across heads. DeepSeek reported that DeepSeek-V2, compared with its earlier 67-billion-parameter dense model, cut the KV cache by 93.3% and raised maximum generation throughput 5.76-fold, and that MLA matched or beat standard multi-head attentionmulti-head attention, n.: Running several attention processes side by side, each free to track a different kind of relationship between words — grammar, topic, who did what — and then combining their results. in its ablations. It is used in DeepSeek-V2, V3 and R1 and in later models such as Moonshot AI's Kimi K2 (2025).
With MLA the cache per token is a 512-dimensional latent instead of full keys and values for every head.
Origin
Introduced by DeepSeek-AI in the DeepSeek-V2 technical report (May 2024) and retained in DeepSeek-V3 (December 2024), whose cost-efficiency drew wide attention in early 2025.
Related
- multi-head attentionmulti-head attention, n.: Running several attention processes side by side, each free to track a different kind of relationship between words — grammar, topic, who did what — and then combining their results.
- grouped-query attentiongrouped-query attention, n.: A middle ground between full multi-head attention and its cheapest variant: attention heads are split into groups, and each group shares one store of information about earlier words, saving memory with little loss in quality.
- multi-query attentionmulti-query attention, n.: A cheaper form of attention in which all the attention heads share one store of information about earlier words instead of each keeping its own, making text generation faster and lighter on memory.
- KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch.
- low-rank approximationlow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation.
- mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
Sources
- DeepSeek-AI (2024). DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model. arXiv:2405.04434. arxiv.org/abs/2405.04434
Mentioned in
attentionattention, n.: A way for an AI model to choose what to focus on. For each word it processes, the model scores how relevant every other word is and draws information mostly from the relevant ones., DeepSeekDeepSeek, prop. n.: A Chinese artificial intelligence company whose free, cheaply trained models startled the world in January 2025 by nearly matching the best American systems., KV cacheKV cache, n.: A memory store where a chatbot keeps its working notes on everything already read and written, so each new word does not require re-reading the whole conversation from scratch., low-rank approximationlow-rank approximation, n.: Replacing a big grid of numbers with a slimmer one that keeps most of the same information, the way a few pencil strokes can capture a face. It saves memory and computation., multi-head attentionmulti-head attention, n.: Running several attention processes side by side, each free to track a different kind of relationship between words — grammar, topic, who did what — and then combining their results., grouped-query attentiongrouped-query attention, n.: A middle ground between full multi-head attention and its cheapest variant: attention heads are split into groups, and each group shares one store of information about earlier words, saving memory with little loss in quality., KV cache compressionKV cache compression, n.: Techniques that shrink the memory an AI model uses to remember a long conversation or document, by storing its notes more compactly or discarding the least useful ones., multi-query attentionmulti-query attention, n.: A cheaper form of attention in which all the attention heads share one store of information about earlier words instead of each keeping its own, making text generation faster and lighter on memory.