Distributed data-parallel algorithms aim to accelerate the training of deep
neural networks by parallelizing the computation of large mini-batch gradient
updates across multiple nodes. Approaches that synchronize nodes using exact
distributed averaging (e.g., via AllReduce) are sensitive to stragglers and
communication delays. The PushSum gossip algorithm is robust to these issues,
but only performs approximate distributed averaging. This paper studies
Stochastic Gradient Push (SGP), which combines PushSum with stochastic gradient
updates. We prove that SGP converges to a stationary point of smooth,
non-convex objectives at the same sub-linear rate as SGD, and that all nodes
achieve consensus. We empirically validate the performance of SGP on image
classification (ResNet-50, ImageNet) and machine translation (Transformer,
WMT'16 En-De) workloads. Our code will be made publicly available.

本文研究了 Stochastic Gradient Push 算法，将 PushSum gossip 算法与随机梯度更新相结合，证明 SGP 能以与 SGD 相同的亚线性速率收敛于平稳点，并且所有节点都能达成共识。研究结果在图像分类（ResNet-50，ImageNet）和机器翻译（Transformer，WMT'16 En-De）中进行了经验验证。