Communication-efficient SGD algorithms, which allow nodes to perform local
updates and periodically synchronize local models, are highly effective in
improving the speed and scalability of distributed SGD. However, a rigorous
convergence analysis and comparative study of different communication-reduction
strategies remains a largely open problem. This paper presents a unified
framework called Cooperative SGD that subsumes existing communication-efficient
SGD algorithms such as periodic-averaging, elastic-averaging and decentralized
SGD. By analyzing Cooperative SGD, we provide novel convergence guarantees for
existing algorithms. Moreover, this framework enables us to design new
communication-efficient SGD algorithms that strike the best balance between
reducing communication overhead and achieving fast error convergence with low
error floor.

本文提出了一个统一的框架 —— 合作 SGD (Cooperative SGD)，它包含了现有的高效通信 SGD 算法，如周期平均、弹性平均和分散 SGD。通过分析 Cooperative SGD，我们为现有算法提供了新的收敛保证。此外，该框架使我们能够设计新的高效通信 SGD 算法，以在降低通信开销和实现低误差下的快速误差收敛之间取得最佳平衡。