Cloud computing is becoming increasingly popular as a platform for
distributed training of deep neural networks. Synchronous stochastic gradient
descent (SSGD) suffers from substantial slowdowns due to stragglers if the
environment is non-dedicated, as is common in cloud computing. Asynchronous SGD
(ASGD) methods are immune to these slowdowns but are scarcely used due to
gradient staleness, which encumbers the convergence process. Recent techniques
have had limited success mitigating the gradient staleness when scaling up to
many workers (computing nodes). In this paper we define the Gap as a measure of
gradient staleness and propose Gap-Aware (GA), a novel asynchronous-distributed
method that penalizes stale gradients linearly to the Gap and performs well
even when scaling to large numbers of workers. Our evaluation on the CIFAR,
ImageNet, and WikiText-103 datasets shows that GA outperforms the currently
acceptable gradient penalization method, in final test accuracy. We also
provide convergence rate proof for GA. Despite prior beliefs, we show that if
GA is applied, momentum becomes beneficial in asynchronous environments, even
when the number of workers scales up.

该研究提出了一种名为 Gap-Aware（GA）的新的异步 - 分布式方法，通过线性惩罚梯度滞后度量 Gap 并在大量工作者的情况下表现良好，即使在异步环境中应用 GA，动量仍然有益处。

梯度滞后的间隔感知缓解

Gap Aware Mitigation of Gradient Staleness

Although distributed computing can significantly reduce the training time of
deep neural networks, scaling the training process while maintaining high
efficiency and final accuracy is challenging. Distributed asynchronous training
enjoys near-linear speedup, but asynchrony causes gradient staleness - the main
difficulty in scaling stochastic gradient descent to large clusters. Momentum,
which is often used to accelerate convergence and escape local minima,
exacerbates the gradient staleness, thereby hindering convergence. We propose
DANA: a novel technique for asynchronous distributed SGD with momentum that
mitigates gradient staleness by computing the gradient on an estimated future
position of the model's parameters. Thereby, we show for the first time that
momentum can be fully incorporated in asynchronous training with almost no
ramifications to final accuracy. Our evaluation on the CIFAR and ImageNet
datasets shows that DANA outperforms existing methods, in both final accuracy
and convergence speed while scaling up to a total batch size of 16K on 64
asynchronous workers.

提出了一种名为 DANA 的技术，旨在解决分布式异步训练的梯度走样问题，可以加速深度神经网络的训练过程且不影响最终精度。该方法在 CIFAR 和 ImageNet 数据集上表现出比现有方法更好的性能。