Differential Privacy (DP) can be applied to raw text by exploiting the
spatial arrangement of words in an embedding space. We investigate the
implications of such text privatization on Language Models (LMs) and their
tendency towards stereotypical associations. Since previous studies documented
that linguistic proficiency correlates with stereotypical bias, one could
assume that techniques for text privatization, which are known to degrade
language modeling capabilities, would cancel out undesirable biases. By testing
BERT models trained on texts containing biased statements primed with varying
degrees of privacy, our study reveals that while stereotypical bias generally
diminishes when privacy is tightened, text privatization does not uniformly
equate to diminishing bias across all social domains. This highlights the need
for careful diagnosis of bias in LMs that undergo text privatization.

利用词在嵌入空间中的空间排列对原始文本应用差分隐私 (DP)，研究了此类文本隐私技术对语言模型 (LMs) 及其对刻板化关联的倾向的影响。在测试 BERT 模型在包含有刻板化陈述的文本上进行的训练中，研究发现，尽管隐私越加严格时刻板化倾向一般减弱，但文本隐私并不一致地等同于减少所有社会领域的偏见。这突出了对经历文本隐私处理的语言模型中的偏见进行仔细诊断的必要性。

从保护隐私的预训练中表征刻板偏见

Characterizing Stereotypical Bias from Privacy-preserving Pre-Training

The task of text privatization using Differential Privacy has recently taken
the form of $\textit{text rewriting}$, in which an input text is obfuscated via
the use of generative (large) language models. While these methods have shown
promising results in the ability to preserve privacy, these methods rely on
autoregressive models which lack a mechanism to contextualize the private
rewriting process. In response to this, we propose $\textbf{DP-MLM}$, a new
method for differentially private text rewriting based on leveraging masked
language models (MLMs) to rewrite text in a semantically similar $\textit{and}$
obfuscated manner. We accomplish this with a simple contextualization
technique, whereby we rewrite a text one token at a time. We find that
utilizing encoder-only MLMs provides better utility preservation at lower
$\varepsilon$ levels, as compared to previous methods relying on larger models
with a decoder. In addition, MLMs allow for greater customization of the
rewriting mechanism, as opposed to generative approaches. We make the code for
$\textbf{DP-MLM}$ public and reusable, found at this https URL .

基于掩码语言模型（MLMs），提出了一种新的基于差异隐私的文本重写方法 DP-MLM，能够在语义相似且混淆的方式下重写文本，对于保护隐私提供更好的实用性，并具有更大的重写机制定制化能力。