In this paper, we present a distributed variant of adaptive stochastic
gradient method for training deep neural networks in the parameter-server
model. To reduce the communication cost among the workers and server, we
incorporate two types of quantization schemes, i.e., gradient quantization and
weight quantization, into the proposed distributed Adam. Besides, to reduce the
bias introduced by quantization operations, we propose an error-feedback
technique to compensate for the quantized gradient. Theoretically, in the
stochastic nonconvex setting, we show that the distributed adaptive gradient
method with gradient quantization and error-feedback converges to the
first-order stationary point, and that the distributed adaptive gradient method
with weight quantization and error-feedback converges to the point related to
the quantized level under both the single-worker and multi-worker modes. At
last, we apply the proposed distributed adaptive gradient methods to train deep
neural networks. Experimental results demonstrate the efficacy of our methods.

本文提出一种分布式变体的自适应随机梯度方法用于训练深度神经网络，并通过梯度量化和权重量化等量化方案以及误差反馈技术来降低通信成本和量化误差，实现在随机非凸的场景下达到一阶稳定点，该方法在深度神经网络训练中取得了良好实验结果。