router
In plain wordsIn a mixture-of-experts model, the small component that decides which specialist sub-networks should handle each word, like a dispatcher assigning jobs to workers.
- 1
Mixture of experts
The learned gating component of a mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network. layer that scores every expert for each input tokentoken, n.: The small chunk of text an AI language model reads and writes — often a word, part of a word or a punctuation mark. AI usage, limits and prices are counted in tokens. — typically a linear layer followed by a softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score. or sigmoid — and selects the top-k experts (commonly k = 1, 2 or 8), whose outputs are weighted by the normalized scores. Because selection is discrete, the router learns only through the weights of the experts it chose, and it tends to collapse onto a few favored experts unless counteracted: Switch Transformer added an auxiliary load-balancing loss, other designs add noise or cap each expert's capacity and drop or reroute overflow tokens, and DeepSeek-V3 balances load mainly by adjusting per-expert bias terms, keeping only a very small sequence-level auxiliary loss. In expert-choice routing, experts pick tokens rather than the reverse. Router instability and imbalance across devices are frequent causes of MoE training problems.
Router logits are computed in float32, because bfloat16 routing caused the loss to spike.
- 2
Applications
A component that directs each whole request to one of several models, endpoints or tools — for example, sending easy queries to a cheap model and hard ones to a stronger model; see model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality. and AI gatewayAI gateway, n.: A checkpoint between an organization's apps and the AI models they use that handles access, costs, logging, safety checks and switching between providers in one place.. Unlike an MoE router, it works outside any single model.
Origin
Gating networks date to the 1991 mixture-of-experts work of Robert Jacobs, Michael Jordan, Steven Nowlan and Geoffrey Hinton. 'Router' and 'routing' became the usual terms for token-level gating with GShard (2020) and Switch Transformer (William Fedus, Barret Zoph and Noam Shazeer, 2021), which sent each token to a single expert.
Related
- mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network.
- active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests.
- expert parallelismexpert parallelism, n.: A way of running a 'mixture of experts' model — one made of many specialist sub-networks — by giving each chip its own set of specialists and sending each piece of text to whichever chip holds the specialists it needs.
- softmaxsoftmax, n.: A formula that turns a list of raw scores into probabilities — all positive and adding up to one — with the biggest share going to the highest score.
- model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality.
- feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge.
Sources
- Fedus, W., Zoph, B. & Shazeer, N. (2021). Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity. arXiv:2101.03961. arxiv.org/abs/2101.03961
- DeepSeek-AI (2024). DeepSeek-V3 Technical Report. arXiv:2412.19437. arxiv.org/abs/2412.19437
Mentioned in
active parametersactive parameters, n.: The part of a model's internal numbers actually used to process each word. Some big models use only a small slice of themselves at a time, making them cheaper to run than their size suggests., feed-forward networkfeed-forward network, n.: The part of each transformer layer that works on every word separately, after attention has mixed in context. It holds most of a model's adjustable numbers and is thought to store much of its knowledge., mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network., model routingmodel routing, n.: Automatically choosing which AI model should answer each request, a cheap quick one for easy questions and a bigger slower one for hard ones, to save money without hurting quality., expert parallelismexpert parallelism, n.: A way of running a 'mixture of experts' model — one made of many specialist sub-networks — by giving each chip its own set of specialists and sending each piece of text to whichever chip holds the specialists it needs., sparse upcyclingsparse upcycling, n.: Recycling an already-trained AI model into a larger 'team of specialists' model by copying parts of it, so the new model starts from what the old one learned instead of from scratch., z-lossz-loss, n.: A small extra penalty added during training that stops the raw scores a language model produces before choosing a word from growing huge, which helps keep very large training runs from blowing up.