Privacy preservation remains a key challenge in data mining and Natural
Language Understanding (NLU). Previous research shows that the input text or
even text embeddings can leak private information. This concern motivates our
research on effective privacy preservation approaches for pretrained Language
Models (LMs). We investigate the privacy and utility implications of applying
dx-privacy, a variant of Local Differential Privacy, to BERT fine-tuning in NLU
applications. More importantly, we further propose privacy-adaptive LM
pretraining methods and show that our approach can boost the utility of BERT
dramatically while retaining the same level of privacy protection. We also
quantify the level of privacy preservation and provide guidance on privacy
configuration. Our experiments and findings lay the groundwork for future
explorations of privacy-preserving NLU with pretrained LMs.

在自然语言理解和数据挖掘中，隐私保护仍然是一个关键的挑战。本文研究了对预先训练的语言模型应用局部差分隐私的隐私保护方法，并指出了这种方法在 BERT 细调中的实用性和隐私性的影响，同时提出了隐私适应性的语言模型预训练方法，可以显著提高 BERT 的实用性，并保持相同水平的隐私保护。本文实验和发现为未来探索隐私保护的自然语言理解提供了奠基。