With the increase in the amount of data and the expansion of model scale, distributed parallel training becomes an important and successful technique to address the optimization challenges. Nevertheless, although distributed stochastic gradient descent (SGD) algorithms can achieve a linear iteration speedup, they are limited significantly in practice by the communication cost, making it difficult to achieve a linear time speedup. In this paper, we propose a computation and communication decoupled stochastic gradient descent (CoCoD-SGD) algorithm to run computation and communication in parallel to reduce the communication cost. We prove that CoCoD-SGD has a linear iteration speedup with respect to the total computation capability of the hardware resources. In addition, it has a lower communication complexity and better time speedup comparing with traditional distributed SGD algorithms. Experiments on deep neural network training demonstrate the significant improvements of CoCoD-SGD: when training ResNet18 and VGG16 with 16 Geforce GTX 1080Ti GPUs, CoCoD-SGD is up to 2-3$\times$ faster than traditional synchronous SGD.

本文提出了Computation and Communication Decoupling Stochastic Gradient Descent (CoCoD-SGD)算法, 实现了计算和通信的并行处理, 有效减少了通信开销, 较传统分布式SGD算法具有更高的时间加速度, 在16个GPU上的ResNet18和VGG16深度神经网络训练表现出2-3倍的速度提升。

更快的分布式深度神经网络训练：计算和通信解耦合的随机梯度下降