Decentralized training of deep learning models is a key element for enabling
data privacy and on-device learning over networks, as well as for efficient
scaling to large compute clusters. As current approaches suffer from limited
bandwidth of the network, we propose the use of communication compression in
the decentralized training context. We show that Choco-SGD $-$ recently
introduced and analyzed for strongly-convex objectives only $-$ converges under
arbitrary high compression ratio on general non-convex functions at the rate
$O\bigl(1/\sqrt{nT}\bigr)$ where $T$ denotes the number of iterations and $n$
the number of workers. The algorithm achieves linear speedup in the number of
workers and supports higher compression than previous state-of-the art methods.
We demonstrate the practical performance of the algorithm in two key scenarios:
the training of deep learning models (i) over distributed user devices,
connected by a social network and (ii) in a datacenter (outperforming
all-reduce time-wise).

该论文研究了去中心化训练深度学习模型的关键因素，提出使用通信压缩来解决网络带宽有限的问题，并通过 Choco-SGD 算法实现了高压缩下的快速收敛，同时支持更高压缩比例，实验结果表明在多种应用场景下具有很好的实际性能表现。