Differentially Private Stochastic Gradient Descent with gradient clipping
(DPSGD-GC) is a powerful tool for training deep learning models using sensitive
data, providing both a solid theoretical privacy guarantee and high efficiency.
However, using DPSGD-GC to ensure Differential Privacy (DP) comes at the cost
of model performance degradation due to DP noise injection and gradient
clipping. Existing research has extensively analyzed the theoretical
convergence of DPSGD-GC, and has shown that it only converges when using large
clipping thresholds that are dependent on problem-specific parameters.
Unfortunately, these parameters are often unknown in practice, making it hard
to choose the optimal clipping threshold. Therefore, in practice, DPSGD-GC
suffers from degraded performance due to the {\it constant} bias introduced by
the clipping.
In our work, we propose a new error-feedback (EF) DP algorithm as an
alternative to DPSGD-GC, which not only offers a diminishing utility bound
without inducing a constant clipping bias, but more importantly, it allows for
an arbitrary choice of clipping threshold that is independent of the problem.
We establish an algorithm-specific DP analysis for our proposed algorithm,
providing privacy guarantees based on R{\'e}nyi DP. Additionally, we
demonstrate that under mild conditions, our algorithm can achieve nearly the
same utility bound as DPSGD without gradient clipping. Our empirical results on
Cifar-10/100 and E2E datasets, show that the proposed algorithm achieves higher
accuracies than DPSGD while maintaining the same level of DP guarantee.

我们提出了一种新的误差反馈（EF）DP 算法作为 DPSGD-GC 的替代方案，它不仅提供了逐渐减小的效用界限而且不引入恒定的剪裁偏差，更重要的是，它允许独立于问题进行剪裁阈值的任意选择。

无剪切偏差的差分隐私 SGD：一种误差反馈方法

Differentially Private SGD Without Clipping Bias: An Error-Feedback  Approach

Training differentially private machine learning models requires constraining
an individual's contribution to the optimization process. This is achieved by
clipping the $2$-norm of their gradient at a predetermined threshold prior to
averaging and batch sanitization. This selection adversely influences
optimization in two opposing ways: it either exacerbates the bias due to
excessive clipping at lower values, or augments sanitization noise at higher
values. The choice significantly hinges on factors such as the dataset, model
architecture, and even varies within the same optimization, demanding
meticulous tuning usually accomplished through a grid search. In order to
circumvent the privacy expenses incurred in hyperparameter tuning, we present a
novel approach to dynamically optimize the clipping threshold. We treat this
threshold as an additional learnable parameter, establishing a clean
relationship between the threshold and the cost function. This allows us to
optimize the former with gradient descent, with minimal repercussions on the
overall privacy analysis. Our method is thoroughly assessed against alternative
fixed and adaptive strategies across diverse datasets, tasks, model dimensions,
and privacy levels. Our results demonstrate its comparable or superior
performance in all evaluated scenarios, given the same privacy requirements.

通过将剪切阈值视为可学习参数，我们提出了一种新的方法来动态优化差分隐私机器学习模型的训练过程，从而在不牺牲整体隐私分析的情况下，在各种评估场景中展现出与传统方法相媲美甚至更好的性能。

差分隐私学习中的在线敏感性优化

Online Sensitivity Optimization in Differentially Private Learning

Gradient clipping is a popular modification to standard (stochastic) gradient
descent, at every iteration limiting the gradient norm to a certain value $c
>0$. It is widely used for example for stabilizing the training of deep
learning models (Goodfellow et al., 2016), or for enforcing differential
privacy (Abadi et al., 2016). Despite popularity and simplicity of the clipping
mechanism, its convergence guarantees often require specific values of $c$ and
strong noise assumptions.
In this paper, we give convergence guarantees that show precise dependence on
arbitrary clipping thresholds $c$ and show that our guarantees are tight with
both deterministic and stochastic gradients. In particular, we show that (i)
for deterministic gradient descent, the clipping threshold only affects the
higher-order terms of convergence, (ii) in the stochastic setting convergence
to the true optimum cannot be guaranteed under the standard noise assumption,
even under arbitrary small step-sizes. We give matching upper and lower bounds
for convergence of the gradient norm when running clipped SGD, and illustrate
these results with experiments.

本文研究了梯度裁剪在随机梯度下降中的应用，给出了裁剪阈值对收敛结果的影响和其上下界，进一步阐述了裁剪机制的缺陷及解决方案。