Recently, there has been an increase in efforts to understand how large
language models (LLMs) propagate and amplify social biases. Several works have
utilized templates for fairness evaluation, which allow researchers to quantify
social biases in the absence of test sets with protected attribute labels.
While template evaluation can be a convenient and helpful diagnostic tool to
understand model deficiencies, it often uses a simplistic and limited set of
templates. In this paper, we study whether bias measurements are sensitive to
the choice of templates used for benchmarking. Specifically, we investigate the
instability of bias measurements by manually modifying templates proposed in
previous works in a semantically-preserving manner and measuring bias across
these modifications. We find that bias values and resulting conclusions vary
considerably across template modifications on four tasks, ranging from an 81%
reduction (NLI) to a 162% increase (MLM) in (task-specific) bias measurements.
Our results indicate that quantifying fairness in LLMs, as done in current
practice, can be brittle and needs to be approached with more care and caution.

研究证明，使用样板评估来量化社会偏见可能不可靠，本文研究了在不破坏语义的情况下手动修改先前工作中使用的模板是否会影响偏见测量的结果，结果显示在四项任务中，模板的修改会使偏见值和相应的结论在不同程度上变化，有增有减，因此，对于 LLMs 的公平性评估需要更加谨慎。