Bias research in NLP seeks to analyse models for social biases, thus helping
NLP practitioners uncover, measure, and mitigate social harms. We analyse the
body of work that uses prompts and templates to assess bias in language models.
We draw on a measurement modelling framework to create a taxonomy of attributes
that capture what a bias test aims to measure and how that measurement is
carried out. By applying this taxonomy to 90 bias tests, we illustrate
qualitatively and quantitatively that core aspects of bias test
conceptualisations and operationalisations are frequently unstated or
ambiguous, carry implicit assumptions, or be mismatched. Our analysis
illuminates the scope of possible bias types the field is able to measure, and
reveals types that are as yet under-researched. We offer guidance to enable the
community to explore a wider section of the possible bias space, and to better
close the gap between desired outcomes and experimental design, both for bias
and for evaluating language models more broadly.

通过量化和定性分析 90 个偏见测试，本文在应用测量建模框架来创造捕捉偏见测试目标的属性分类法的基础上，揭示了偏见测试的核心概念和操作存在的不明确、含蓄、或不匹配的偏差种类及其频度，并提出指导方案，以便更全面地探索偏见空间和更全面地评估语言模型的实验设计。