Distributed adaptive stochastic gradient methods have been widely used for
large-scale nonconvex optimization, such as training deep learning models.
However, their communication complexity on finding $\varepsilon$-stationary
points has rarely been analyzed in the nonconvex setting. In this work, we
present a novel communication-efficient distributed Adam in the
parameter-server model for stochastic nonconvex optimization, dubbed {\em
Efficient-Adam}. Specifically, we incorporate a two-way quantization scheme
into Efficient-Adam to reduce the communication cost between the workers and
server. Simultaneously, we adopt a two-way error feedback strategy to reduce
the biases caused by the two-way quantization on both the server and workers,
respectively. In addition, we establish the iteration complexity for the
proposed Efficient-Adam with a class of quantization operators, and further
characterize its communication complexity between the server and workers when
an $\varepsilon$-stationary point is achieved. Finally, we apply Efficient-Adam
to solve a toy stochastic convex optimization problem and train deep learning
models on real-world vision and language tasks. Extensive experiments together
with a theoretical guarantee justify the merits of Efficient Adam.

本文提出了一种称为 Efficient-Adam 的新型通信高效的分布式优化算法，采用双向量化策略和双向误差反馈策略，在保证迭代收敛精度的同时有效减少了服务器与工作节点之间的通信成本，并应用于解决了一个随机凸优化问题，以及在实际视觉和语言任务中训练深度学习模型，同时进行了大量实验和理论分析以验证其优越性。