The vanilla Differentially-Private Stochastic Gradient Descent (DP-SGD),
including DP-Adam and other variants, ensures the privacy of training data by
uniformly distributing privacy costs across training steps. The equivalent
privacy costs controlled by maintaining the same gradient clipping thresholds
and noise powers in each step result in unstable updates and a lower model
accuracy when compared to the non-DP counterpart. In this paper, we propose the
dynamic DP-SGD (along with dynamic DP-Adam, and others) to reduce the
performance loss gap while maintaining privacy by dynamically adjusting
clipping thresholds and noise powers while adhering to a total privacy budget
constraint. Extensive experiments on a variety of deep learning tasks,
including image classification, natural language processing, and federated
learning, demonstrate that the proposed dynamic DP-SGD algorithm stabilizes
updates and, as a result, significantly improves model accuracy in the strong
privacy protection region when compared to the vanilla DP-SGD. We also conduct
theoretical analysis to better understand the privacy-utility trade-off with
dynamic DP-SGD, as well as to learn why Dynamic DP-SGD can outperform vanilla
DP-SGD.

本文提出了动态 DP-SGD 算法，通过动态调整剪裁阈值和噪声幅度来降低性能损失，同时保持隐私，从而显著提高了模型的准确性。