Decentralized optimization is emerging as a viable alternative for scalable
distributed machine learning, but also introduces new challenges in terms of
synchronization costs. To this end, several communication-reduction techniques,
such as non-blocking communication, quantization, and local steps, have been
explored in the decentralized setting. Due to the complexity of analyzing
optimization in such a relaxed setting, this line of work often assumes
\emph{global} communication rounds, which require additional synchronization.
In this paper, we consider decentralized optimization in the simpler, but
harder to analyze, \emph{asynchronous gossip} model, in which communication
occurs in discrete, randomly chosen pairings among nodes. Perhaps surprisingly,
we show that a variant of SGD called \emph{SwarmSGD} still converges in this
setting, even if \emph{non-blocking communication}, \emph{quantization}, and
\emph{local steps} are all applied \emph{in conjunction}, and even if the node
data distributions and underlying graph topology are both \emph{heterogenous}.
Our analysis is based on a new connection with multi-dimensional load-balancing
processes. We implement this algorithm and deploy it in a super-computing
environment, showing that it can outperform previous decentralized methods in
terms of end-to-end training time, and that it can even rival carefully-tuned
large-batch SGD for certain tasks.

研究异步传播模型下去中心化优化算法的收敛性，提出一种称为 SwarmSGD 的变量 SGD 算法，使用多维负载平衡过程连接进行分析，并在超级计算环境中表现出比以前的去中心化算法和大批次 SGD 更好的性能。