Large language models (LLMs) can pass explicit bias tests but still harbor
implicit biases, similar to humans who endorse egalitarian beliefs yet exhibit
subtle biases. Measuring such implicit biases can be a challenge: as LLMs
become increasingly proprietary, it may not be possible to access their
embeddings and apply existing bias measures; furthermore, implicit biases are
primarily a concern if they affect the actual decisions that these systems
make. We address both of these challenges by introducing two measures of bias
inspired by psychology: LLM Implicit Association Test (IAT) Bias, which is a
prompt-based method for revealing implicit bias; and LLM Decision Bias for
detecting subtle discrimination in decision-making tasks. Using these measures,
we found pervasive human-like stereotype biases in 6 LLMs across 4 social
domains (race, gender, religion, health) and 21 categories (weapons, guilt,
science, career among others). Our prompt-based measure of implicit bias
correlates with embedding-based methods but better predicts downstream
behaviors measured by LLM Decision Bias. This measure is based on asking the
LLM to decide between individuals, motivated by psychological results
indicating that relative not absolute evaluations are more related to implicit
biases. Using prompt-based measures informed by psychology allows us to
effectively expose nuanced biases and subtle discrimination in proprietary LLMs
that do not show explicit bias on standard benchmarks.

通过心理学启发的 LLM 暗示联想测试偏差和 LLM 决策偏差等两项偏差测量方法，揭示了大规模语言模型中普遍存在的人类化刻板印象偏差，以及对决策任务中的微妙歧视进行检测。

在显式无偏的大型语言模型中测量隐性偏见

Measuring Implicit Bias in Explicitly Unbiased Large Language Models

Warning: This paper contains several contents that may be toxic, harmful, or
offensive.
In the last few years, text-to-image generative models have gained remarkable
success in generating images with unprecedented quality accompanied by a
breakthrough of inference speed. Despite their rapid progress, human biases
that manifest in the training examples, particularly with regard to common
stereotypical biases, like gender and skin tone, still have been found in these
generative models. In this work, we seek to measure more complex human biases
exist in the task of text-to-image generations. Inspired by the well-known
Implicit Association Test (IAT) from social psychology, we propose a novel
Text-to-Image Association Test (T2IAT) framework that quantifies the implicit
stereotypes between concepts and valence, and those in the images. We replicate
the previously documented bias tests on generative models, including morally
neutral tests on flowers and insects as well as demographic stereotypical tests
on diverse social attributes. The results of these experiments demonstrate the
presence of complex stereotypical behaviors in image generations.

本研究在探讨文本转图像生成模型领域的人类偏见及刻板印象问题，并为此提出了一种基于社会心理学中的隐性联系测验（Implicit Association Test）的文本到图像联系测试框架（Text-to-Image Association Test），通过实验验证模型在道德中性和民族刻板印象等各方面存在复杂的刻板印象行为。