Language models are widely deployed to provide automatic text completion
services in user products. However, recent research has revealed that language
models (especially large ones) bear considerable risk of memorizing private
training data, which is then vulnerable to leakage and extraction by
adversaries. In this study, we test the efficacy of a range of
privacy-preserving techniques to mitigate unintended memorization of sensitive
user text, while varying other factors such as model size and adversarial
conditions. We test both "heuristic" mitigations (those without formal privacy
guarantees) and Differentially Private training, which provides provable levels
of privacy at the cost of some model performance. Our experiments show that
(with the exception of L2 regularization), heuristic mitigations are largely
ineffective in preventing memorization in our test suite, possibly because they
make too strong of assumptions about the characteristics that define
"sensitive" or "private" text. In contrast, Differential Privacy reliably
prevents memorization in our experiments, despite its computational and
model-performance costs.

通过测试效用一系列隐私保护技术来缓解大型语言模型不经意地记忆保密训练数据的风险，实验表明，差分隐私有可靠的隐私保护效果，但是性能代价较大。