mixture of experts
In plain wordsA way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
An architecture in which a layer contains many parallel sub-networks ('experts') and a learned routerrouter, n.: In a mixture-of-experts model, the small component that decides which specialist sub-networks should handle each word, like a dispatcher assigning jobs to workers. sends each input — in LLMslarge language model, n.: A computer program trained on enormous amounts of text to predict the words that come next. At vast scale, that simple skill lets it chat, answer questions, translate, summarize and write code., each tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. at each MoE layer — to only a few of them, combining their outputs with the router's weights. In transformers, experts usually replace the feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge. in some or all blocks. Because only k of E experts run per token, total parameters can grow far faster than compute per token (active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests.): Mixtral 8x7B has about 47 billion parameters but uses about 13 billion per token, and DeepSeek-V3 has 671 billion with 37 billion active. Engineering challenges include balancing load across experts, overflow when an expert receives too many tokens, the communication cost of expert parallelismexpert parallelism, n.: A way of running a 'mixture of experts' model — one made of many specialist sub-networks — by giving each chip its own set of specialists and sending each piece of text to whichever chip holds the specialists it needs., and memory to hold every expert. Fine-grained designs use many small experts plus always-active shared experts. Most leading open-weight LLMs released in 2025 were MoEs, as are some documented proprietary models such as Gemini 1.5.
The MoE has 128 experts per layer and routes each token to eight of them.
Origin
Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton introduced adaptive mixtures of local experts in 1991. Noam Shazeer and colleagues at Google revived the idea at scale with the sparsely-gated MoE layer (January 2017); GShard (2020) and Switch Transformer (2021) brought it to transformers, and Mixtral (December 2023) and DeepSeek (2024–2025) made it standard in open models.
Usage note
Experts rarely specialize in human-recognizable topics; routing mostly reflects token-level and syntactic patterns. 'Sparse' refers to sparse activation, not sparse weights. Unlike an ensemble, an MoE runs only a few members per input.
Related
- routerrouter, n.: In a mixture-of-experts model, the small component that decides which specialist sub-networks should handle each word, like a dispatcher assigning jobs to workers.
- active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests.
- feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge.
- expert parallelismexpert parallelism, n.: A way of running a 'mixture of experts' model — one made of many specialist sub-networks — by giving each chip its own set of specialists and sending each piece of text to whichever chip holds the specialists it needs.
- transformertransformer, n.: The design behind today's chatbots and much of modern AI. It handles text by letting every word look at every other word and decide which of them matter for understanding it.
- parameter countparameter count, n.: How many adjustable numbers an AI model contains — the usual measure of its size. Modern language models have billions to trillions of them.
- sparse attentionsparse attention, n.: Attention in which each word looks at only a selected subset of the other words instead of all of them, so very long texts can be processed at a fraction of the cost.
- ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model.
Sources
- Jacobs, R. A., Jordan, M. I., Nowlan, S. J. & Hinton, G. E. (1991). Adaptive Mixtures of Local Experts. Neural Computation 3(1).
- Shazeer, N., Mirhoseini, A., Maziarz, K., Davis, A., Le, Q., Hinton, G. & Dean, J. (2017). Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer. ICLR 2017. arxiv.org/abs/1701.06538
- Fedus, W., Zoph, B. & Shazeer, N. (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. JMLR 23 (2022). arxiv.org/abs/2101.03961
Mentioned in
DeepSeekDeepSeek, prop. n.: A Chinese artificial intelligence company whose free, cheaply trained models startled the world in January 2025 by nearly matching the best American systems., GeminiGemini, prop. n.: Google's family of AI models and its chatbot assistant, launched in December 2023. Gemini models handle text, images, audio and video and power many Google products., LlamaLlama, prop. n.: A family of AI language models from Meta, first released in 2023, whose trained settings can be downloaded and run by anyone, making them the foundation for thousands of other AI projects., loss functionloss function, n.: A score that measures how wrong a model's predictions are. Training means adjusting the model, step by step, to push this number down., scaling lawsscaling laws, n.: Measured patterns showing that an AI model's error shrinks smoothly and predictably as the model is made bigger and given more data and computing power, so labs can forecast a giant model's performance from small, cheap experiments., active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests., architecturearchitecture, n.: The overall design of an AI model, meaning what kinds of parts it has and how they are connected, as opposed to the specific numbers it learns during training., data mixturedata mixture, n.: The recipe of different kinds of data a model is trained on — how much web text, code, books, math, science and other languages — which strongly shapes what the finished model is good at., decoder-only transformerdecoder-only transformer, n.: The kind of transformer behind most chatbots: a single stack that reads text left to right and continues it, without a separate part that first encodes an input., DeepSeek-R1DeepSeek-R1, prop. n.: A free, downloadable AI reasoning model released by the Chinese company DeepSeek in January 2025. It matched leading American models at math and coding for a fraction of the reported cost, shaking the industry., ensemble learningensemble learning, n.: Combining the predictions of several models — by voting or averaging — so their individual mistakes partly cancel out, giving more accurate and stable results than any one model., feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge., Google BrainGoogle Brain, prop. n.: Google's deep-learning research team from 2011 to 2023, where the transformer — the design behind today's chatbots — and the TensorFlow software were created., gpt-ossgpt-oss, prop. n.: Two free-to-download reasoning models released by OpenAI in August 2025 — its first downloadable language models since 2019 — that anyone may run, modify and use commercially on their own hardware., GrokGrok, prop. n.: The AI chatbot and model family of Elon Musk's company xAI, launched in 2023 and built into the social network X. It is known for its irreverent style and for repeated controversies over its outputs., Liang WenfengLiang Wenfeng, prop. n.: A Chinese entrepreneur who founded DeepSeek, whose low-cost, openly released AI models shocked global markets in January 2025., Mistral AIMistral AI, prop. n.: Europe's best-known builder of large AI language models, a Paris start-up dating from 2023 whose chatbot is called Le Chat. Many of its models are free for anyone to download and run., model capacitymodel capacity, n.: How much a model is able to learn: the variety and complexity of patterns it can represent or memorize. Too little and it misses real patterns; very large models can even memorize pure noise., model mergingmodel merging, n.: Combining two or more trained AI models into one by blending their internal numbers directly, so the result shares their skills without any further training., model parallelismmodel parallelism, n.: Splitting a single AI model that is too big for one chip into pieces spread over several chips, which must constantly pass intermediate results to each other as they work., model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality., Moonshot AIMoonshot AI, prop. n.: A Beijing artificial intelligence company, founded in 2023, that makes the Kimi chatbot and very large models that anyone can download., multi-agent systemmulti-agent system, n.: A setup in which several AI agents, each with its own role or point of view, work together — or compete — on a problem, passing messages and results among themselves., Noam ShazeerNoam Shazeer, prop. n.: An American computer scientist who co-invented the transformer and several techniques found in nearly every large language model, co-founded Character.AI, and later returned to Google to co-lead Gemini.