The superior performance of large foundation models relies on the use of
massive amounts of high-quality data, which often contain sensitive, private
and copyrighted material that requires formal protection. While differential
privacy (DP) is a prominent method to gauge the degree of security provided to
the models, its application is commonly limited to the model fine-tuning stage,
due to the performance degradation when applying DP during the pre-training
stage. Consequently, DP is yet not capable of protecting a substantial portion
of the data used during the initial pre-training process.
In this work, we first provide a theoretical understanding of the efficacy of
DP training by analyzing the per-iteration loss improvement. We make a key
observation that DP optimizers' performance degradation can be significantly
mitigated by the use of limited public data, which leads to a novel DP
continual pre-training strategy. Empirically, using only 10\% of public data,
our strategy can achieve DP accuracy of 41.5\% on ImageNet-21k (with
$\epsilon=8$), as well as non-DP accuracy of 55.7\% and and 60.0\% on
downstream tasks Places365 and iNaturalist-2021, respectively, on par with
state-of-the-art standard pre-training and substantially outperforming existing
DP pre-trained models.

通过使用有限的公共数据，我们提出了一种新颖的差分隐私持续预训练策略，可以显著减轻差分隐私优化器的性能下降问题，并在 ImageNet-21k 上实现 41.5% 的差分隐私准确率（ε=8），以及在下游任务 Places365 和 iNaturalist-2021 上分别达到 55.7% 和 60.0% 的非差分隐私准确率，与当前最先进的标准预训练方法相媲美并且明显优于现有的差分隐私预训练模型。

具有有限公共数据的差分隐私模型的预训练

Pre-training Differentially Private Models with Limited Public Data

Differential privacy (DP) is the de facto standard for training machine
learning (ML) models, including neural networks, while ensuring the privacy of
individual examples in the training set. Despite a rich literature on how to
train ML models with differential privacy, it remains extremely challenging to
train real-life, large neural networks with both reasonable accuracy and
privacy.
We set out to investigate how to do this, using ImageNet image classification
as a poster example of an ML task that is very challenging to resolve
accurately with DP right now. This paper shares initial lessons from our
effort, in the hope that it will inspire and inform other researchers to
explore DP training at scale. We show approaches that help make DP training
faster, as well as model types and settings of the training process that tend
to work better in the DP setting. Combined, the methods we discuss let us train
a Resnet-18 with DP to $47.9\%$ accuracy and privacy parameters $\epsilon = 10,
\delta = 10^{-6}$. This is a significant improvement over "naive" DP training
of ImageNet models, but a far cry from the $75\%$ accuracy that can be obtained
by the same network without privacy. The model we use was pretrained on the
Places365 data set as a starting point. We share our code at
this https URL, calling for others to build
upon this new baseline to further improve DP at scale.

本文旨在探究如何使用差分隐私来训练机器学习模型，以 ImageNet 图像分类为例，展示如何使用方法和模型类型来让训练过程更好地进行。我们展示了一些方法，使我们能够使用 DP 来训练一个 ResNet-18，精度为 47.9％。虽然这比 “朴素” 的 DP 训练要好，但是离没有隐私的情况下的 75％精度还有很大的差距。