By ensuring differential privacy in the learning algorithms, one can
rigorously mitigate the risk of large models memorizing sensitive training
data. In this paper, we study two algorithms for this purpose, i.e., DP-SGD and
DP-NSGD, which first clip or normalize \textit{per-sample} gradients to bound
the sensitivity and then add noise to obfuscate the exact information. We
analyze the convergence behavior of these two algorithms in the non-convex
optimization setting with two common assumptions and achieve a rate
$\mathcal{O}\left(\sqrt[4]{\frac{d\log(1/\delta)}{N^2\epsilon^2}}\right)$ of
the gradient norm for a $d$-dimensional model, $N$ samples and
$(\epsilon,\delta)$-DP, which improves over previous bounds under much weaker
assumptions. Specifically, we introduce a regularizing factor in DP-NSGD and
show that it is crucial in the convergence proof and subtly controls the bias
and noise trade-off. Our proof deliberately handles the per-sample gradient
clipping and normalization that are specified for the private setting.
Empirically, we demonstrate that these two algorithms achieve similar best
accuracy while DP-NSGD is comparatively easier to tune than DP-SGD and hence
may help further save the privacy budget when accounting the tuning effort.

本文探讨了确保差分隐私的两个算法 DP-SGD 和 DP-NSGD，并在非凸优化设定下分析了这两种算法的收敛行为及其梯度范数的速度，同时介绍了 DP-NSGD 的正则化因子如何控制偏差和噪声的平衡。