The rapid advancement of large language models has revolutionized various
applications but also raised crucial concerns about their potential to
perpetuate biases and unfairness when deployed in social media contexts.
Evaluating LLMs' potential biases and fairness has become crucial, as existing
methods rely on limited prompts focusing on just a few groups, lacking a
comprehensive categorical perspective. In this paper, we propose evaluating LLM
biases from a group fairness lens using a novel hierarchical schema
characterizing diverse social groups. Specifically, we construct a dataset,
GFair, encapsulating target-attribute combinations across multiple dimensions.
In addition, we introduce statement organization, a new open-ended text
generation task, to uncover complex biases in LLMs. Extensive evaluations of
popular LLMs reveal inherent safety concerns. To mitigate the biases of LLM
from a group fairness perspective, we pioneer a novel chain-of-thought method
GF-Think to mitigate biases of LLMs from a group fairness perspective.
Experimental results demonstrate its efficacy in mitigating bias in LLMs to
achieve fairness.

通过使用划分群体公平性的全新层次模式，我们提出了对大型语言模型进行偏见评估的方法，并构建了一个跨多个维度的目标属性组合的数据集，GFair。此外，我们还引入了一项新的开放式文本生成任务来揭示大型语言模型中的复杂偏见。通过对流行的大型语言模型进行广泛评估，我们发现存在固有的安全问题。为了从群体公平性的角度减轻大型语言模型的偏见，我们首创了一种新的思维链 (GF-Think) 方法。实验结果表明，此方法在减轻大型语言模型中的偏见以实现公平性方面非常有效。

大规模语言模型的团体公平透视

A Group Fairness Lens for Large Language Models

We present RadGraph2, a novel dataset for extracting information from
radiology reports that focuses on capturing changes in disease state and device
placement over time. We introduce a hierarchical schema that organizes entities
based on their relationships and show that using this hierarchy during training
improves the performance of an information extraction model. Specifically, we
propose a modification to the DyGIE++ framework, resulting in our model HGIE,
which outperforms previous models in entity and relation extraction tasks. We
demonstrate that RadGraph2 enables models to capture a wider variety of
findings and perform better at relation extraction compared to those trained on
the original RadGraph dataset. Our work provides the foundation for developing
automated systems that can track disease progression over time and develop
information extraction models that leverage the natural hierarchy of labels in
the medical domain.

我们提出了 RadGraph2，这是一个用于从放射学报告中提取信息的新颖数据集，专注于捕捉疾病状态和设备放置随时间变化的情况。我们介绍了一个基于关系组织实体的层次结构模式，并展示使用此层次结构在训练过程中可以提高信息提取模型的性能。具体地，我们提出了对 DyGIE++ 框架的修改，得到我们的模型 HGIE，该模型在实体和关系提取任务中优于先前的模型。我们证明了 RadGraph2 使模型能够捕捉更多种类的发现，并且在关系提取方面表现比原始的 RadGraph 数据集上训练的模型更好。我们的工作为开发可以随时间追踪疾病进展的自动化系统提供了基础，并利用医学领域的自然层级标签来发展信息提取模型。