Deep learning models are increasingly popular in many machine learning
applications where the training data may contain sensitive information. To
provide formal and rigorous privacy guarantee, many learning systems now
incorporate differential privacy by training their models with (differentially)
private SGD. A key step in each private SGD update is gradient clipping that
shrinks the gradient of an individual example whenever its L2 norm exceeds some
threshold. We first demonstrate how gradient clipping can prevent SGD from
converging to stationary point. We then provide a theoretical analysis that
fully quantifies the clipping bias on convergence with a disparity measure
between the gradient distribution and a geometrically symmetric distribution.
Our empirical evaluation further suggests that the gradient distributions along
the trajectory of private SGD indeed exhibit symmetric structure that favors
convergence. Together, our results provide an explanation why private SGD with
gradient clipping remains effective in practice despite its potential clipping
bias. Finally, we develop a new perturbation-based technique that can provably
correct the clipping bias even for instances with highly asymmetric gradient
distributions.

本文研究深度学习中涉及到隐私保护的问题，探讨了梯度裁剪在保证隐私的前提下防止 SGD 算法陷入局部极小值的作用，并提出了一种基于扰动的新技术用于解决梯度分布不对称问题。