Deep learning techniques based on neural networks have shown significant
success in a wide range of AI tasks. Large-scale training datasets are one of
the critical factors for their success. However, when the training datasets are
crowdsourced from individuals and contain sensitive information, the model
parameters may encode private information and bear the risks of privacy
leakage. The recent growing trend of the sharing and publishing of pre-trained
models further aggravates such privacy risks. To tackle this problem, we
propose a differentially private approach for training neural networks. Our
approach includes several new techniques for optimizing both privacy loss and
model accuracy. We employ a generalization of differential privacy called
concentrated differential privacy(CDP), with both a formal and refined privacy
loss analysis on two different data batching methods. We implement a dynamic
privacy budget allocator over the course of training to improve model accuracy.
Extensive experiments demonstrate that our approach effectively improves
privacy loss accounting, training efficiency and model quality under a given
privacy budget.

提出了一种基于集中差分隐私的神经网络训练方法，通过动态隐私预算分配和新的优化技术来提高模型准确性、训练效率和隐私保护能力。