Automatic evaluation metrics are crucial to the development of generative
systems. In recent years, pre-trained language model (PLM) based metrics, such
as BERTScore, have been commonly adopted in various generation tasks. However,
it has been demonstrated that PLMs encode a range of stereotypical societal
biases, leading to a concern on the fairness of PLMs as metrics. To that end,
this work presents the first systematic study on the social bias in PLM-based
metrics. We demonstrate that popular PLM-based metrics exhibit significantly
higher social bias than traditional metrics on 6 sensitive attributes, namely
race, gender, religion, physical appearance, age, and socioeconomic status.
In-depth analysis suggests that choosing paradigms (matching, regression, or
generation) of the metric has a greater impact on fairness than choosing PLMs.
In addition, we develop debiasing adapters that are injected into PLM layers,
mitigating bias in PLM-based metrics while retaining high performance for
evaluating text generation.

本文首次对基于预训练语言模型的度量方法中的社会偏见进行了系统研究，并发现相比传统度量方法，最受欢迎的基于预训练语言模型的度量方法在种族、性别、宗教、外貌、年龄和社会经济地位等 6 个敏感属性上显示出显著的社会偏向。此外，研究发现选择度量方法的范式比选择预训练语言模型对公平性的影响更大，我们提出了消除偏见的适配器方案，将其注入到预训练语言模型层中，从而减轻了基于预训练语言模型的度量方法的偏见，同时保持了评估文本生成的高性能。