With the increase in the amount of data and the expansion of model scale,
distributed parallel training becomes an important and successful technique to
address the optimization challenges. Nevertheless, although distributed
stochastic gradient descent (SGD) algorithms can achieve a linear iteration
speedup, they are limited significantly in practice by the communication cost,
making it difficult to achieve a linear time speedup. In this paper, we propose
a computation and communication decoupled stochastic gradient descent
(CoCoD-SGD) algorithm to run computation and communication in parallel to
reduce the communication cost. We prove that CoCoD-SGD has a linear iteration
speedup with respect to the total computation capability of the hardware
resources. In addition, it has a lower communication complexity and better time
speedup comparing with traditional distributed SGD algorithms. Experiments on
deep neural network training demonstrate the significant improvements of
CoCoD-SGD: when training ResNet18 and VGG16 with 16 Geforce GTX 1080Ti GPUs,
CoCoD-SGD is up to 2-3$\times$ faster than traditional synchronous SGD.

本文提出了 Computation and Communication Decoupling Stochastic Gradient Descent (CoCoD-SGD) 算法，实现了计算和通信的并行处理，有效减少了通信开销，较传统分布式 SGD 算法具有更高的时间加速度，在 16 个 GPU 上的 ResNet18 和 VGG16 深度神经网络训练表现出 2-3 倍的速度提升。