We study the Differential Privacy (DP) guarantee of hidden-state Noisy-SGD algorithms over a bounded domain. Standard privacy analysis for Noisy-SGD assumes all internal states are revealed, which leads to a divergent R'enyi DP bound with respect to the number of iterations. Ye & Shokri (2022) and Altschuler & Talwar (2022) proved convergent bounds for smooth (strongly) convex losses, and raise open questions about whether these assumptions can be relaxed. We provide positive answers by proving convergent R'enyi DP bound for non-convex non-smooth losses, where we show that requiring losses to have H\"older continuous gradient is sufficient. We also provide a strictly better privacy bound compared to state-of-the-art results for smooth strongly convex losses. Our analysis relies on the improvement of shifted divergence analysis in multiple aspects, including forward Wasserstein distance tracking, identifying the optimal shifts allocation, and the H"older reduction lemma. Our results further elucidate the benefit of hidden-state analysis for DP and its applicability.

本文研究了带噪声SGD算法在非凸非平滑损失下的差分隐私保证，填补了该领域的研究空白。通过充分利用Hölder连续梯度的条件，作者证明了收敛的R'enyi差分隐私界限，并提供了相较于已有成果更优的隐私界限。这项研究进一步展示了隐状态分析在差分隐私中的优势及其应用潜力。

非凸和平滑情况下带噪声SGD的收敛隐私损失