Natural language reflects our private lives and identities, making its
privacy concerns as broad as those of real life. Language models lack the
ability to understand the context and sensitivity of text, and tend to memorize
phrases present in their training sets. An adversary can exploit this tendency
to extract training data. Depending on the nature of the content and the
context in which this data was collected, this could violate expectations of
privacy. Thus there is a growing interest in techniques for training language
models that preserve privacy. In this paper, we discuss the mismatch between
the narrow assumptions made by popular data protection techniques (data
sanitization and differential privacy), and the broadness of natural language
and of privacy as a social norm. We argue that existing protection methods
cannot guarantee a generic and meaningful notion of privacy for language
models. We conclude that language models should be trained on text data which
was explicitly produced for public use.

本文讨论保护自然语言隐私的技术是否足够广泛，提出现有数据保护方法不能为语言模型提供通用且有意义的隐私保护，因此应基于明确为公共使用的文本数据对语言模型进行训练。