Distributed machine learning is making great changes in a wide variety of domains but also brings privacy risk from the exchanged information during the learning process. This paper focuses on a class of regularized empirical risk minimization problems, and develops a privacy-preserving distributed learning algorithm. We use Alternating Direction Method of Multipliers (ADMM) to decentralize the learning algorithm, and apply Gaussian mechanisms locally to guarantee differential privacy. However, simply combining ADMM and local randomization mechanisms would result in an unconvergent algorithm with bad performance, especially when the introduced noise is large to guarantee a low total privacy loss. Besides, this approach cannot be applied to the learning problems with non-smooth objective functions. To figure out these concerns, we propose an improved ADMM-based differentially private distributed learning algorithm: DP-ADMM, where an approximate augmented Lagrangian function and Gaussian mechanisms with time-varying variance are utilized. We also apply the moment accountant method to bound the total privacy loss. Our theoretical analysis proves that DP-ADMM can be applied to a general class of convex learning problems, provides differential privacy guarantee, and achieves an $O(1/\sqrt{t})$ rate of convergence, where $t$ is the number of iterations. Our evaluations demonstrate that our approach can achieve good accuracy and effectiveness even with a low total privacy leakage.

本文提供了DP-ADMM算法，它结合了近似增广拉格朗日函数和时间变化的高斯噪声添加，以在相同的不同隐私保证下，实现更高效的一般目标函数的分布式学习，同时应用矩accountant方法来限制端到端的隐私损失。DP-ADMM算法可以用于更广泛的分布式学习问题，具有可证明的收敛性和明确的效用-隐私权衡，并且可以实现高的收敛性和模型精度。

DP-ADMM: 基于ADMM的差分隐私分布式学习