Large Language Models (LLMs) are trained on inherently biased data. Previous
works on debiasing models rely on benchmark datasets to measure model
performance. However, these datasets suffer from several pitfalls due to the
extremely subjective understanding of bias, highlighting a critical need for
contextual exploration. We propose understanding the context of user inputs
with consideration of the diverse situations in which input statements are
possible. This approach would allow for frameworks that foster bias awareness
rather than guardrails that hurt user engagement. Our contribution is twofold:
(i) we create a dataset of 2287 stereotyped statements augmented with points
for adding context; (ii) we develop the Context-Oriented Bias Indicator and
Assessment Score (COBIAS) to assess statements' contextual reliability in
measuring bias. Our metric is a significant predictor of the contextual
reliability of bias-benchmark datasets ($\chi^2=71.02, p<2.2 \cdot 10^{-16})$.
COBIAS can be used to create reliable datasets, resulting in an improvement in
bias mitigation works.

大型语言模型 (LLMs) 的研究及应用中存在偏见问题，本论文提出了一种基于上下文的偏见指标和评估分数 (COBIAS) 来测量偏见的可靠性，并且通过创建可靠的数据集来改善偏见减轻工作。