State-of-the-art distributed machine learning suffers from significant delays due to frequent communication and synchronizing between worker nodes. Emerging communication-efficient SGD algorithms that limit synchronization between locally trained models have been shown to be effective in speeding-up distributed SGD. However, a rigorous convergence analysis and comparative study of different communication-reduction strategies remains a largely open problem. This paper presents a new framework called Coooperative SGD that subsumes existing communication-efficient SGD algorithms such as federated-averaging, elastic-averaging and decentralized SGD. By analyzing Cooperative SGD, we provide novel convergence guarantees for existing algorithms. Moreover this framework enables us to design new communication-efficient SGD algorithms that strike the best balance between reducing communication overhead and achieving fast error convergence.

本文提出了一个统一的框架——合作 SGD (Cooperative SGD)，它包含了现有的高效通信 SGD 算法，如周期平均、弹性平均和分散 SGD。通过分析 Cooperative SGD，我们为现有算法提供了新的收敛保证。此外，该框架使我们能够设计新的高效通信 SGD 算法，以在降低通信开销和实现低误差下的快速误差收敛之间取得最佳平衡。

合作式 SGD: 通信高效 SGD 算法的设计与分析统一框架