State-of-the-art approaches for training Differentially Private (DP) Deep
Neural Networks (DNN) faces difficulties to estimate tight bounds on the
sensitivity of the network's layers, and instead rely on a process of
per-sample gradient clipping. This clipping process not only biases the
direction of gradients but also proves costly both in memory consumption and in
computation. To provide sensitivity bounds and bypass the drawbacks of the
clipping process, our theoretical analysis of Lipschitz constrained networks
reveals an unexplored link between the Lipschitz constant with respect to their
input and the one with respect to their parameters. By bounding the Lipschitz
constant of each layer with respect to its parameters we guarantee DP training
of these networks. This analysis not only allows the computation of the
aforementioned sensitivities at scale but also provides leads on to how
maximize the gradient-to-noise ratio for fixed privacy guarantees. To
facilitate the application of Lipschitz networks and foster robust and
certifiable learning under privacy guarantees, we provide a Python package that
implements building blocks allowing the construction and private training of
such networks.

本文介绍利用限制 Lipschitz 常数的深度神经网络训练，通过对每层的参数进行 Lipschitz 常数约束来提供灵敏度边界，实现差分隐私以保证网络的私密性，在固定的隐私保证下最大化梯度与噪声之比，并提供 Python 包以实现构建和私人训练。

不剪辑的 DP-SGD：利普希茨神经网络之路

DP-SGD Without Clipping: The Lipschitz Neural Network Way

The Gradient Boosting Decision Tree (GBDT) is a popular machine learning
model for various tasks in recent years. In this paper, we study how to improve
model accuracy of GBDT while preserving the strong guarantee of differential
privacy. Sensitivity and privacy budget are two key design aspects for the
effectiveness of differential private models. Existing solutions for GBDT with
differential privacy suffer from the significant accuracy loss due to too loose
sensitivity bounds and ineffective privacy budget allocations (especially
across different trees in the GBDT model). Loose sensitivity bounds lead to
more noise to obtain a fixed privacy level. Ineffective privacy budget
allocations worsen the accuracy loss especially when the number of trees is
large. Therefore, we propose a new GBDT training algorithm that achieves
tighter sensitivity bounds and more effective noise allocations. Specifically,
by investigating the property of gradient and the contribution of each tree in
GBDTs, we propose to adaptively control the gradients of training data for each
iteration and leaf node clipping in order to tighten the sensitivity bounds.
Furthermore, we design a novel boosting framework to allocate the privacy
budget between trees so that the accuracy loss can be further reduced. Our
experiments show that our approach can achieve much better model accuracy than
other baselines.

本篇论文旨在研究如何提高 Gradient Boosting Decision Tree 模型的准确性，同时保持差分隐私强有力的保护措施。通过改进灵敏度限制和与树之间隐私预算的分配算法，该论文提出一种更有效的差分隐私 GBDT 训练算法，以实现更高的模型准确性。