In decentralized machine learning, workers compute model updates on their
local data. Because the workers only communicate with few neighbors without
central coordination, these updates propagate progressively over the network.
This paradigm enables distributed training on networks without all-to-all
connectivity, helping to protect data privacy as well as to reduce the
communication cost of distributed training in data centers. A key challenge,
primarily in decentralized deep learning, remains the handling of differences
between the workers' local data distributions. To tackle this challenge, we
introduce the RelaySum mechanism for information propagation in decentralized
learning. RelaySum uses spanning trees to distribute information exactly
uniformly across all workers with finite delays depending on the distance
between nodes. In contrast, the typical gossip averaging mechanism only
distributes data uniformly asymptotically while using the same communication
volume per step as RelaySum. We prove that RelaySGD, based on this mechanism,
is independent of data heterogeneity and scales to many workers, enabling
highly accurate decentralized deep learning on heterogeneous data. Our code is
available at this http URL

本研究提出了 RelaySum 机制来解决分散式学习中不同工作节点之间数据分布的差异所带来的挑战，该机制利用生成树将信息分配给所有工作节点，并证明了基于该机制的 RelaySGD 对于数据异构性是独立的，并可扩展到许多工作节点，从而在异构数据上实现高精度的分散式深度学习。