Large language models have been shown to encode a variety of social biases,
which carries the risk of downstream harms. While the impact of these biases
has been recognized, prior methods for bias evaluation have been limited to
binary association tests on small datasets, offering a constrained view of the
nature of societal biases within language models. In this paper, we propose an
original framework for probing language models for societal biases. We collect
a probing dataset to analyze language models' general associations, as well as
along the axes of societal categories, identities, and stereotypes. To this
end, we leverage a novel perplexity-based fairness score. We curate a
large-scale benchmarking dataset addressing drawbacks and limitations of
existing fairness collections, expanding to a variety of different identities
and stereotypes. When comparing our methodology with prior work, we demonstrate
that biases within language models are more nuanced than previously
acknowledged. In agreement with recent findings, we find that larger model
variants exhibit a higher degree of bias. Moreover, we expose how identities
expressing different religions lead to the most pronounced disparate treatments
across all models.

本研究提出了一种新的框架，用于探索语言模型中的社会偏见，通过采集探索数据集和利用一种新的公平性评分方法，发现语言模型中的偏见更加复杂，并揭示不同宗教身份导致各种模型中最明显的不平等处理。

社会偏见探测：语言模型的公平性基准测试

Social Bias Probing: Fairness Benchmarking for Language Models

In this work, we tackle the scenario of understanding characters in scripts,
which aims to learn the characters' personalities and identities from their
utterances. We begin by analyzing several challenges in this scenario, and then
propose a multi-level contrastive learning framework to capture characters'
global information in a fine-grained manner. To validate the proposed
framework, we conduct extensive experiments on three character understanding
sub-tasks by comparing with strong pre-trained language models, including
SpanBERT, Longformer, BigBird and ChatGPT-3.5. Experimental results demonstrate
that our method improves the performances by a considerable margin. Through
further in-depth analysis, we show the effectiveness of our method in
addressing the challenges and provide more hints on the scenario of character
understanding. We will open-source our work on github at
this https URL

我们提出了一个多级对比学习框架来捕捉角色的全局信息，通过与强大的预训练语言模型进行比较，在角色理解的三个子任务中我们的方法显著提高了性能，并通过进一步深入分析展示了方法的有效性和对角色理解场景的指导意义。

多级对比学习用于基于字母的字符理解

Multi-level Contrastive Learning for Script-based Character  Understanding

Intelligent conversational agents, or chatbots, can take on various
identities and are increasingly engaging in more human-centered conversations
with persuasive goals. However, little is known about how identities and
inquiry strategies influence the conversation's effectiveness. We conducted an
online study involving 790 participants to be persuaded by a chatbot for
charity donation. We designed a two by four factorial experiment (two chatbot
identities and four inquiry strategies) where participants were randomly
assigned to different conditions. Findings showed that the perceived identity
of the chatbot had significant effects on the persuasion outcome (i.e.,
donation) and interpersonal perceptions (i.e., competence, confidence, warmth,
and sincerity). Further, we identified interaction effects among perceived
identities and inquiry strategies. We discuss the findings for theoretical and
practical implications for developing ethical and effective persuasive
chatbots. Our published data, codes, and analyses serve as the first step
towards building competent ethical persuasive chatbots.

本研究通过一个 2 x 4 的实验，在 790 名参与者中测试了聊天机器人的身份认同和质询策略对于慈善捐赠的影响，并发现其身份认同对于说服结果和人际感知有着显著性影响，还进一步探讨了身份认同和质询策略之间的交互作用，旨在为发展具有伦理和有效性的聊天机器人提供理论和实践上的启示。