Distributed training algorithms of deep neural networks show impressive
convergence speedup properties on very large problems. However, they inherently
suffer from communication related slowdowns and communication topology becomes
a crucial design choice. Common approaches supported by most machine learning
frameworks are: 1) Synchronous decentralized algorithms relying on a
peer-to-peer All Reduce topology that is sensitive to stragglers and
communication delays. 2) Asynchronous centralised algorithms with a server
based topology that is prone to communication bottleneck. Researchers also
suggested asynchronous decentralized algorithms designed to avoid the
bottleneck and speedup training, however, those commonly use inexact sparse
averaging that may lead to a degradation in accuracy. In this paper, we propose
Local Asynchronous SGD (LASGD), an asynchronous decentralized algorithm that
relies on All Reduce for model synchronization.
We empirically validate LASGD's performance on image classification tasks on
the ImageNet dataset. Our experiments demonstrate that LASGD accelerates
training compared to SGD and state of the art gossip based approaches.

本文主要介绍分布式深度神经网络训练算法的通信拓扑设计选择及异步去中心化算法如何通过 LASGD 实现模型同步，实验证明 LASGD 相较于 SGD 及业界领先的基于八卦协议的算法加速了大规模图像分类数据集 ImageNet 的训练速度。

去中心化深度学习的本地异步随机梯度下降

Locally Asynchronous Stochastic Gradient Descent for Decentralised Deep Learning

Many areas of deep learning benefit from using increasingly larger neural
networks trained on public data, as is the case for pre-trained models for NLP
and computer vision. Training such models requires a lot of computational
resources (e.g., HPC clusters) that are not available to small research groups
and independent researchers. One way to address it is for several smaller
groups to pool their computational resources together and train a model that
benefits all participants. Unfortunately, in this case, any participant can
jeopardize the entire training run by sending incorrect updates, deliberately
or by mistake. Training in presence of such peers requires specialized
distributed training algorithms with Byzantine tolerance. These algorithms
often sacrifice efficiency by introducing redundant communication or passing
all updates through a trusted server, making it infeasible to apply them to
large-scale deep learning, where models can have billions of parameters. In
this work, we propose a novel protocol for secure (Byzantine-tolerant)
decentralized training that emphasizes communication efficiency.

本文提出了一种新颖的协议，用于安全的（拜占庭容错）分散式训练，强调通信效率。