In modern large-scale machine learning applications, the training data are often partitioned and stored on multiple machines. It is customary to employ the "data parallelism" approach, where the aggregated training loss is minimized without moving data across machines. In this paper, we introduce a novel distributed dual formulation for regularized loss minimization problems that can directly handle data parallelism under the distributed computing environment. This formulation allows us to systematically derive dual coordinate optimization procedures, which we refer to as Distributed Alternating Dual Maximization (DADM). The method extends earlier studies about distributed SDCA algorithms and has a rigorous theoretical analysis. Based on the new formulation, we also develop an accelerated DADM algorithm by generalizing the acceleration technique from the accelerated SDCA algorithm to the distributed setting. Our empirical studies show that our novel approach significantly improves the previous state-of-the-art distributed dual coordinate optimization algorithms.

本文提出了一个新的分布式双重正则化损失函数最小化问题的方法，可以直接处理数据并行性，并允许系统地导出二次坐标优化过程。通过这个新的公式，我们开发了分布式替代双重最大化（DADM）的加速版本，并提供了理论上的性能分析，发现其性能明显优于以前的分布式坐标优化算法。

一种通用的分布式双坐标优化框架用于正则化损失最小化