Rapid advances in Natural Language Processing (NLP) have revolutionized many fields, including healthcare. However, these advances raise significant privacy concerns, especially when models specialized on sensitive data can memorize and then expose and regurgitate confidential information. This paper presents a privacy-by-design language modeling approach to address the problem of language models anonymization, and thus promote their sharing. Specifically, we propose both a Masking Language Modeling (MLM) methodology to specialize a BERT-like language model, and a Causal Language Modeling (CLM) methodology to specialize a GPT-like model that avoids the model from memorizing direct and indirect identifying information present in the training data. We have comprehensively evaluated our approaches using medical datasets and compared them against different baselines. Our results indicate that by avoiding memorizing both direct and indirect identifiers during model specialization, our masking and causal language modeling schemes offer the best tradeoff for maintaining high privacy while retaining high utility.

该研究解决了语言模型在处理敏感数据时可能暴露隐私信息的问题。论文提出了一种隐私设计的语言建模方法，包括遮蔽语言建模和因果语言建模，旨在防止模型记住训练数据中的识别信息。研究结果表明，这些方法在保持高隐私性的同时，也能保持高效用性。

从设计角度进行语言建模的匿名化