The widespread use of large language models has brought up essential
questions about the potential biases these models might learn. This led to the
development of several metrics aimed at evaluating and mitigating these biases.
In this paper, we first demonstrate that prompt-based fairness metrics exhibit
poor agreement, as measured by correlation, raising important questions about
the reliability of fairness assessment using prompts. Then, we outline six
relevant reasons why such a low correlation is observed across existing
metrics. Based on these insights, we propose a method called Correlated
Fairness Output (CAIRO) to enhance the correlation between fairness metrics.
CAIRO augments the original prompts of a given fairness metric by using several
pre-trained language models and then selects the combination of the augmented
prompts that achieves the highest correlation across metrics. We show a
significant improvement in Pearson correlation from 0.3 and 0.18 to 0.90 and
0.98 across metrics for gender and religion biases, respectively. Our code is
available at this https URL

该研究针对大型语言模型可能学习到的偏见，通过评估和减轻这些偏见的多个指标发现，基于提示的公平性指标间的相关性较低，因此提出了一种名为 CAIRO 的方法，通过增强提示信息来提高公平性指标之间的相关性，并取得了显著的改善。