sparse upcycling

A way to build a sparse mixture of expertsmixture of experts, n.: A way to build very large AI models cheaply: the model contains many specialist sub-networks, and for each word only a few are switched on, picked by a small dispatcher network. (MoE) model by converting a trained dense model, so that it starts from that checkpointcheckpoint, n.: A saved snapshot of a model partway through or at the end of training, like a save point in a video game: training can resume from it after a crash, and finished snapshots are what people download and run.'s weights rather than random ones. Selected feed-forward (MLP) layers of the dense network are replicated to form the experts, a newly initialized routerrouter, n.: In a mixture-of-experts model, the small component that decides which specialist sub-networks should handle each word, like a dispatcher assigning jobs to workers. is inserted to send each token to a few of them, and all other weights (attention, embeddings, normalization) are copied unchanged; training then continues with the MoE architecture. Because the copied experts start identical, they must diverge during further training, and variants such as partially re-initializing the copies ('drop-upcycling') encourage specialization. This avoids wasting an expensive dense run: in the original T5 and Vision Transformer experiments, extra training worth a fraction of the original compute put upcycled models ahead of dense models given the same budget and of MoE models trained from scratch on comparable or larger budgets. It has been used to build released MoE language models, for example Qwen1.5-MoE, upcycled from a 1.8-billion-parameter dense model.

Rather than pretrain a new MoE, we upcycled our 7B checkpoint into eight experts and trained for another 300 billion tokens.