Social bias in Pretrained Language Models (PLMs) affects text generation and other downstream NLP tasks. Existing bias testing methods rely predominantly on manual templates or on expensive crowd-sourced data. We propose a novel AutoBiasTest method that automatically generates sentences for testing bias in PLMs, hence providing a flexible and low-cost alternative. Our approach uses another PLM for generation and controls the generation of sentences by conditioning on social group and attribute terms. We show that generated sentences are natural and similar to human-produced content in terms of word length and diversity. We illustrate that larger models used for generation produce estimates of social bias with lower variance. We find that our bias scores are well correlated with manual templates, but AutoBiasTest highlights biases not captured by these templates due to more diverse and realistic test sentences. By automating large-scale test sentence generation, we enable better estimation of underlying bias distributions

本研究提出了一种名为AutoBiasTest的新方法，可自动生成句子以测试预训练语言模型中的社会偏见，从而提供了一种灵活且低成本的替代方案，通过使用另一个PLM进行生成并在社交群体和属性术语的限制下控制句子的生成。我们展示了生成的句子在词长和多样性方面与人类生成的内容相似，通过自动化大规模测试句子的生成，我们可以更好地估计潜在的偏差分布。 

AutoBiasTest：语言模型中可控的句子生成用于自动和开放式的社会偏见测试