We propose an adaptive (stochastic) gradient perturbation method for
differentially private empirical risk minimization. At each iteration, the
random noise added to the gradient is optimally adapted to the stepsize; we
name this process adaptive differentially private (ADP) learning. Given the
same privacy budget, we prove that the ADP method considerably improves the
utility guarantee compared to the standard differentially private method in
which vanilla random noise is added. Our method is particularly useful for
gradient-based algorithms with time-varying learning rates, including variants
of AdaGrad (Duchi et al., 2011). We provide extensive numerical experiments to
demonstrate the effectiveness of the proposed adaptive differentially private
algorithm.

本文提出了一种自适应的（随机）梯度扰动方法用于差分隐私经验风险最小化，在每次迭代中将添加的随机噪声优化适应于步长；我们将这个过程称为自适应差分隐私（ADP）学习。 通过相同的隐私预算，我们证明了 ADP 方法相比于加入香草随机噪音的标准差分隐私方法，可以显著提高实用保证性能。 我们的方法特别适用于具有时间变化学习率的基于梯度的算法，包括 AdaGrad（Duchi 等，2011）的变体。 我们进行了大量的数字实验，以展示所提出的自适应差分隐私算法的有效性。