Differentially Private (DP) learning has seen limited success for building
large deep learning models of text, and straightforward attempts at applying
Differentially Private Stochastic Gradient Descent (DP-SGD) to NLP tasks have
resulted in large performance drops and high computational overhead. We show
that this performance drop can be mitigated with (1) the use of large
pretrained language models; (2) non-standard hyperparameters that suit DP
optimization; and (3) fine-tuning objectives which are aligned with the
pretraining procedure. With the above, we obtain NLP models that outperform
state-of-the-art DP-trained models under the same privacy budget and strong
non-private baselines -- by directly fine-tuning pretrained models with DP
optimization on moderately-sized corpora. To address the computational
challenge of running DP-SGD with large Transformers, we propose a memory saving
technique that allows clipping in DP-SGD to run without instantiating
per-example gradients for any linear layer in the model. The technique enables
privately training Transformers with almost the same memory cost as non-private
training at a modest run-time overhead. Contrary to conventional wisdom that DP
optimization fails at learning high-dimensional models (due to noise that
scales with dimension) empirical results reveal that private learning with
pretrained language models doesn't tend to suffer from dimension-dependent
performance degradation. Code to reproduce results can be found at
this https URL

使用预训练的语言模型以及非标准化超参数和 fine-tuning 目标结合 DP 优化技术，可在中等规模的语料库上获得胜过强基线和同一隐私预算下的 DP-trained 模型的 NLP 模型。我们还提出了一种内存节省技术来解决在大型 Transformers 上运行 DP-SGD 的计算难题，该技术可以使得 clip 在 DP-SGD 中运行而无需对模型中的任何线性层实例化每个样本的梯度，成本与非隐私的训练相当，并且有适度的运行时间开销。