The Noisy-SGD algorithm is widely used for privately training machine
learning models. Traditional privacy analyses of this algorithm assume that the
internal state is publicly revealed, resulting in privacy loss bounds that
increase indefinitely with the number of iterations. However, recent findings
have shown that if the internal state remains hidden, then the privacy loss
might remain bounded. Nevertheless, this remarkable result heavily relies on
the assumption of (strong) convexity of the loss function. It remains an
important open problem to further relax this condition while proving similar
convergent upper bounds on the privacy loss. In this work, we address this
problem for DP-SGD, a popular variant of Noisy-SGD that incorporates gradient
clipping to limit the impact of individual samples on the training process. Our
findings demonstrate that the privacy loss of projected DP-SGD converges
exponentially fast, without requiring convexity or smoothness assumptions on
the loss function. In addition, we analyze the privacy loss of regularized
(unprojected) DP-SGD. To obtain these results, we directly analyze the
hockey-stick divergence between coupled stochastic processes by relying on
non-linear data processing inequalities.

本文研究了 DP-SGD 算法在限制梯度影响的条件下，对于具有省略凸性和平滑性假设的损失函数，随着迭代次数的增加，其隐私泄露的收敛速度是指数级的。同时，文章还分析了非正则 DP-SGD 的隐私损失。