Plenty of works have brought social biases in language models to attention
and proposed methods to detect such biases. As a result, the literature
contains a great deal of different bias tests and scores, each introduced with
the premise to uncover yet more biases that other scores fail to detect. What
severely lacks in the literature, however, are comparative studies that analyse
such bias scores and help researchers to understand the benefits or limitations
of the existing methods. In this work, we aim to close this gap for cosine
based bias scores. By building on a geometric definition of bias, we propose
requirements for bias scores to be considered meaningful for quantifying
biases. Furthermore, we formally analyze cosine based scores from the
literature with regard to these requirements. We underline these findings with
experiments to show that the bias scores' limitations have an impact in the
application case.

在这项工作中，我们通过基于几何定义的偏见，提出了对偏见得分的要求，以便被认为是衡量偏见的有意义的方法，并通过对文献中的余弦偏见得分进行形式分析，加以强调，并通过实验证明偏见得分的局限性对应用案例产生影响。