In distributed training of deep neural networks, parallel mini-batch SGD is
widely used to speed up the training process by using multiple workers. It uses
multiple workers to sample local stochastic gradient in parallel, aggregates
all gradients in a single server to obtain the average, and update each
worker's local model using a SGD update with the averaged gradient. Ideally,
parallel mini-batch SGD can achieve a linear speed-up of the training time
(with respect to the number of workers) compared with SGD over a single worker.
However, such linear scalability in practice is significantly limited by the
growing demand for gradient communication as more workers are involved. Model
averaging, which periodically averages individual models trained over parallel
workers, is another common practice used for distributed training of deep
neural networks since (Zinkevich et al. 2010) (McDonald, Hall, and Mann 2010).
Compared with parallel mini-batch SGD, the communication overhead of model
averaging is significantly reduced. Impressively, tremendous experimental works
have verified that model averaging can still achieve a good speed-up of the
training time as long as the averaging interval is carefully controlled.
However, it remains a mystery in theory why such a simple heuristic works so
well. This paper provides a thorough and rigorous theoretical study on why
model averaging can work as well as parallel mini-batch SGD with significantly
less communication overhead.

本篇论文探讨了为何模型平均可以降低神经网络分布式训练中的通信开销，并阐述该方法与并行小批量随机梯度下降（SGD）在训练时间上的相似性。