In this pioneering work we formulate ExpM+NF, a method for training machine
learning (ML) on private data with pre-specified differentially privacy
guarantee $\varepsilon>0, \delta=0$, by using the Exponential Mechanism (ExpM)
and an auxiliary Normalizing Flow (NF). We articulate theoretical benefits of
ExpM+NF over Differentially Private Stochastic Gradient Descent (DPSGD), the
state-of-the-art (SOTA) and de facto method for differentially private ML, and
we empirically test ExpM+NF against DPSGD using the SOTA implementation (Opacus
with PRV accounting) in multiple classification tasks on the Adult Dataset
(census data) and MIMIC-III Dataset (electronic healthcare records) using
Logistic Regression and GRU-D, a deep learning recurrent neural network with
~20K-100K parameters. In all experiments, ExpM+NF achieves greater than 93% of
the non-private training accuracy (AUC) for $\varepsilon \in [1\mathrm{e}{-3},
1]$, exhibiting greater accuracy (higher AUC) and privacy (lower $\varepsilon$
with $\delta=0$) than DPSGD. Differentially private ML generally considers
$\varepsilon \in [1,10]$ to maintain reasonable accuracy; hence, ExpM+NF's
ability to provide strong accuracy for orders of magnitude better privacy
(smaller $\varepsilon$) substantially pushes what is currently possible in
differentially private ML. Training time results are presented showing ExpM+NF
is comparable to (slightly faster) than DPSGD. Code for these experiments will
be provided after review. Limitations and future directions are provided.

我们提出了 ExpM+NF 方法，它通过使用指数机制（ExpM）和辅助归一化流（NF）来在私有数据上训练具有预定差分隐私保证的机器学习（ML）模型。我们在多个分类任务上通过 Logistic 回归和 GRU-D 进行实验，ExpM+NF 实现了非私有训练准确率的 93% 以上，比 DPSGD 具有更高的准确性和更好的隐私保护能力。此工作对差分私有 ML 的可能性进行了实质性的推进，并提供了该方法的局限性和未来方向。