In this paper, we propose a differentially private decentralized learning
method (termed PrivSGP-VR) which employs stochastic gradient push with variance
reduction and guarantees $(\epsilon, \delta)$-differential privacy (DP) for
each node. Our theoretical analysis shows that, under DP Gaussian noise with
constant variance, PrivSGP-VR achieves a sub-linear convergence rate of
$\mathcal{O}(1/\sqrt{nK})$, where $n$ and $K$ are the number of nodes and
iterations, respectively, which is independent of stochastic gradient variance,
and achieves a linear speedup with respect to $n$. Leveraging the moments
accountant method, we further derive an optimal $K$ to maximize the model
utility under certain privacy budget in decentralized settings. With this
optimized $K$, PrivSGP-VR achieves a tight utility bound of $\mathcal{O}\left(
\sqrt{d\log \left( \frac{1}{\delta} \right)}/(\sqrt{n}J\epsilon) \right)$,
where $J$ and $d$ are the number of local samples and the dimension of decision
variable, respectively, which matches that of the server-client distributed
counterparts, and exhibits an extra factor of $1/\sqrt{n}$ improvement compared
to that of the existing decentralized counterparts, such as A(DP)$^2$SGD.
Extensive experiments corroborate our theoretical findings, especially in terms
of the maximized utility with optimized $K$, in fully decentralized settings.

提出一种差分隐私的分布式学习方法 (PrivSGP-VR)，利用随机梯度推进和方差缩减，保证每个节点的 (ε, δ)- 差分隐私，并在分布式设置中获得了亚线性收敛率，并实现与节点数成线性加速的效果。通过使用矩阵法进行优化，获得了最优化的迭代次数 K，以最大化模型的效用。实验结果验证了理论发现，尤其是在完全分布式设置中，通过优化 K 实现了最大化的效用。

PrivSGP-VR：差分隐私的方差减小随机梯度推送与紧致效用界

PrivSGP-VR: Differentially Private Variance-Reduced Stochastic Gradient  Push with Tight Utility Bounds

Cooperative decentralized deep learning relies on direct information exchange
between communicating agents, each with access to a local dataset which should
be kept private. The goal is for all agents to achieve consensus on model
parameters after training. However, sharing parameters with untrustworthy
neighboring agents could leak exploitable information about local datasets. To
combat this, we introduce differentially private decentralized learning that
secures each agent's local dataset during and after cooperative training. In
our approach, we generalize Differentially Private Stochastic Gradient Descent
(DP-SGD) -- a popular differentially private training method for centralized
deep learning -- to practical subgradient- and ADMM-based decentralized
learning methods. Our algorithms' differential privacy guarantee holds for
arbitrary deep learning objective functions, and we analyze the convergence
properties for strongly convex objective functions. We compare our algorithms
against centrally trained models on standard classification tasks and evaluate
the relationships between performance, privacy budget, graph connectivity, and
degree of training data overlap among agents. We find that differentially
private gradient tracking is resistant to performance degradation under sparse
graphs and non-uniform data distributions. Furthermore, we show that it is
possible to learn a model achieving high accuracies, within 3% of DP-SGD on
MNIST under (1, 10^-5)-differential privacy and within 6% of DP-SGD on
CIFAR-100 under (10, 10^-5)-differential privacy, without ever sharing raw data
with other agents. Open source code can be found at:
this https URL

本研究提出了一种基于不同隐私的分散式深度学习方法，该方法对协作学习的数据进行隐私保护，保障节点之间数据的安全交换。实验结果表明，在稀疏图和不均匀数据分布下，不同隐私梯度跟踪具有抗干扰性，并且可在不向其他代理共享原始数据的情况下学习高准确性模型。