LLMs are increasingly powerful and widely used to assist users in a variety
of tasks. This use risks the introduction of LLM biases to consequential
decisions such as job hiring, human performance evaluation, and criminal
sentencing. Bias in NLP systems along the lines of gender and ethnicity has
been widely studied, especially for specific stereotypes (e.g., Asians are good
at math). In this paper, we investigate bias along less studied, but still
consequential, dimensions, such as age and beauty, measuring subtler correlated
decisions that LLMs (specially autoregressive language models) make between
social groups and unrelated positive and negative attributes. We ask whether
LLMs hold wide-reaching biases of positive or negative sentiment for specific
social groups similar to the ``what is beautiful is good'' bias found in people
in experimental psychology. We introduce a template-generated dataset of
sentence completion tasks that asks the model to select the most appropriate
attribute to complete an evaluative statement about a person described as a
member of a specific social group. We also reverse the completion task to
select the social group based on an attribute. Finally, we report the
correlations that we find for multiple cutting-edge LLMs. This dataset can be
used as a benchmark to evaluate progress in more generalized biases and the
templating technique can be used to expand the benchmark with minimal
additional human annotation.

通过使用模板生成的数据集，本文研究了 LLMs 在年龄和美貌等维度上的偏见，以及 LLMs 对特定社会群体的情感偏向。同时，我们报告了多个先进的 LLMs 找到的相关性，这个数据集可以用来评估更广泛的偏见，并且模板技术可用于在最小的人工注释下扩展该基准。