When the data is distributed across multiple servers, lowering the
communication cost between the servers (or workers) while solving the
distributed learning problem is an important problem and is the focus of this
paper. In particular, we propose a fast, and communication-efficient
decentralized framework to solve the distributed machine learning (DML)
problem. The proposed algorithm, Group Alternating Direction Method of
Multipliers (GADMM) is based on the Alternating Direction Method of Multipliers
(ADMM) framework. The key novelty in GADMM is that it solves the problem in a
decentralized topology where at most half of the workers are competing for the
limited communication resources at any given time. Moreover, each worker
exchanges the locally trained model only with two neighboring workers, thereby
training a global model with a lower amount of communication overhead in each
exchange. We prove that GADMM converges to the optimal solution for convex loss
functions, and numerically show that it converges faster and more
communication-efficient than the state-of-the-art communication-efficient
algorithms such as the Lazily Aggregated Gradient (LAG) and dual averaging, in
linear and logistic regression tasks on synthetic and real datasets.
Furthermore, we propose Dynamic GADMM (D-GADMM), a variant of GADMM, and prove
its convergence under the time-varying network topology of the workers.

本文提出一种快速且高效的分布式机器学习框架，名为 Group Alternating Direction Method of Multipliers (GADMM)，该算法在分布式 topology 中解决问题，且与现有算法相比，实现更快的收敛和更高的通讯效率；此外，还提出了 GADMM 的变体，即 Dynamic GADMM (D-GADMM)，并证明了其在时间变化的网络拓扑下的收敛性。