Skip to content
Last Dictionary
Plane
Training & Post-trainingHardware & ComputeSociety, Ethics & Economy

decentralized training

nounAlso: decentralised training, collaborative training, volunteer training, decentralized SGD, gossip averaging

In plain wordsTraining an AI model on computers owned by many different people or companies around the world, linked over the ordinary internet rather than housed together in one data center.

  1. 1

    Open-network trainingemerging

    Training a single model on compute contributed by independent, geographically dispersed providers connected over the public internet, often permissionlessly, instead of within one data center. Internet links are orders of magnitude slower and less reliable than data-center interconnectsinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one., so such systems rely on low-communication algorithms (DiLoCoDiLoCo, prop. n.: A training method from Google DeepMind that lets groups of chips in different places train one language model while talking to each other hundreds of times less often than usual., local SGDlocal SGD, n.: A way to train on many machines that talk to each other only occasionally: each trains its own copy for a while, then they average their copies and continue, cutting the amount of communication needed., compressed or quantized updates), on fault tolerance for nodes that join and leave, and sometimes on cryptographic or economic mechanisms to verify honest work. A landmark is Prime Intellect's INTELLECT-1 (2024), a 10-billion-parameter model trained on 1 trillion tokens by up to 14 concurrent nodes on three continents from 30 compute providers; decentralized reinforcement-learning runs followed in 2025. Supporters present it as a counterweight to concentrated compute; governance researchers note it could complicate compute governancecompute governance, n.: Using control over the computer chips and data centers that AI depends on — who can buy them, where they go, how they are used — as a way to oversee powerful AI.. As of 2026 its largest runs remain far smaller than frontier pretraining runs.

    The decentralized training run lost a third of its nodes overnight and kept going.

  2. 2

    Optimization

    Distributed optimization without a central coordinator, in which workers average parameters only with neighbors in a communication graph (gossip averaging), as in decentralized parallel SGD (Xiangru Lian and colleagues, NeurIPS 2017); it removes the bottleneck of a central server or global all-reduce.

Origin

Volunteer deep-learning efforts such as Learning@home and the Hivemind library (Max Ryabinin and Anton Gusev, 2020) and DeDLOC (Michael Diskin and colleagues, 2021) preceded the 2024–2025 wave led by Prime Intellect, Nous Research and others. The coordinator-free optimization sense dates from Lian and colleagues' 2017 paper.

Related

See also
  • DiLoCoDiLoCo, prop. n.: A training method from Google DeepMind that lets groups of chips in different places train one language model while talking to each other hundreds of times less often than usual.
  • local SGDlocal SGD, n.: A way to train on many machines that talk to each other only occasionally: each trains its own copy for a while, then they average their copies and continue, cutting the amount of communication needed.
  • distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks.
  • federated learningfederated learning, n.: Training one shared AI model across many phones, hospitals or companies without collecting their data in one place: each trains on its own data and sends back only model updates.
  • compute governancecompute governance, n.: Using control over the computer chips and data centers that AI depends on — who can buy them, where they go, how they are used — as a way to oversee powerful AI.
  • interconnectinterconnect, n.: The wiring and networking that connect chips to each other — inside a server, across a data center, or within a single chip — so they can share data and work as one.
Contrast with
  • federated learningfederated learning, n.: Training one shared AI model across many phones, hospitals or companies without collecting their data in one place: each trains on its own data and sends back only model updates.

Sources

  1. Jaghouar, S. et al. (2024). INTELLECT-1 Technical Report. arXiv:2412.01152. arxiv.org/abs/2412.01152
  2. Lian, X., Zhang, C., Zhang, H., Hsieh, C.-J., Zhang, W. & Liu, J. (2017). Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descent. NeurIPS 2017.

Mentioned in

distributed trainingdistributed training, n.: Training one AI model on many computer chips at once, splitting up the data or the model itself among them and keeping them in sync, so that jobs that would take a single chip thousands of years finish in weeks., DiLoCoDiLoCo, prop. n.: A training method from Google DeepMind that lets groups of chips in different places train one language model while talking to each other hundreds of times less often than usual., local SGDlocal SGD, n.: A way to train on many machines that talk to each other only occasionally: each trains its own copy for a while, then they average their copies and continue, cutting the amount of communication needed.