Distributed training algorithms of deep neural networks show impressive
convergence speedup properties on very large problems. However, they inherently
suffer from communication related slowdowns and communication topology becomes
a crucial design choice. Common approaches supported by most machine learning
frameworks are: 1) Synchronous decentralized algorithms relying on a
peer-to-peer All Reduce topology that is sensitive to stragglers and
communication delays. 2) Asynchronous centralised algorithms with a server
based topology that is prone to communication bottleneck. Researchers also
suggested asynchronous decentralized algorithms designed to avoid the
bottleneck and speedup training, however, those commonly use inexact sparse
averaging that may lead to a degradation in accuracy. In this paper, we propose
Local Asynchronous SGD (LASGD), an asynchronous decentralized algorithm that
relies on All Reduce for model synchronization.
We empirically validate LASGD's performance on image classification tasks on
the ImageNet dataset. Our experiments demonstrate that LASGD accelerates
training compared to SGD and state of the art gossip based approaches.

本文主要介绍分布式深度神经网络训练算法的通信拓扑设计选择及异步去中心化算法如何通过 LASGD 实现模型同步，实验证明 LASGD 相较于 SGD 及业界领先的基于八卦协议的算法加速了大规模图像分类数据集 ImageNet 的训练速度。