Distributed machine learning has been widely studied in order to handle
exploding amount of data. In this paper, we study an important yet less visited
distributed learning problem where features are inherently distributed or
vertically partitioned among multiple parties, and sharing of raw data or model
parameters among parties is prohibited due to privacy concerns. We propose an
ADMM sharing framework to approach risk minimization over distributed features,
where each party only needs to share a single value for each sample in the
training process, thus minimizing the data leakage risk. We establish
convergence and iteration complexity results for the proposed parallel ADMM
algorithm under non-convex loss. We further introduce a novel differentially
private ADMM sharing algorithm and bound the privacy guarantee with carefully
designed noise perturbation. The experiments based on a prototype system shows
that the proposed ADMM algorithms converge efficiently in a robust fashion,
demonstrating advantage over gradient based methods especially for data set
with high dimensional feature spaces.

本文提出了一种用于处理分布式学习中固有分布或纵向分区的特征的风险最小化方法的 ADMM 共享框架，介绍了新颖的差分隐私 ADMM 共享算法，并通过精心设计的噪声扰动对隐私保证进行了约束，其实验结果表明所提出的 ADMM 算法具有高效的收敛性和鲁棒性，并在高维特征空间的数据集上表现出优势，尤其是相对于梯度方法。

分布式特征隐私学习：ADMM 共享方法

Learning Privately over Distributed Features: An ADMM Sharing Approach

Alternating direction method of multiplier (ADMM) is a popular method used to
design distributed versions of a machine learning algorithm, whereby local
computations are performed on local data with the output exchanged among
neighbors in an iterative fashion. During this iterative process the leakage of
data privacy arises. A differentially private ADMM was proposed in prior work
(Zhang & Zhu, 2017) where only the privacy loss of a single node during one
iteration was bounded, a method that makes it difficult to balance the tradeoff
between the utility attained through distributed computation and privacy
guarantees when considering the total privacy loss of all nodes over the entire
iterative process. We propose a perturbation method for ADMM where the
perturbed term is correlated with the penalty parameters; this is shown to
improve the utility and privacy simultaneously. The method is based on a
modified ADMM where each node independently determines its own penalty
parameter in every iteration and decouples it from the dual updating step size.
The condition for convergence of the modified ADMM and the lower bound on the
convergence rate are also derived.

为了解决分布式计算过程中数据隐私泄露的问题，提出了一种基于改进的 ADMM 算法的扰动方法，可同时提高实用性和隐私性，并得到收敛性条件和收敛速率下界。