Large language models (LLMs) are increasingly being used in human-centered
social scientific tasks, such as data annotation, synthetic data creation, and
engaging in dialog. However, these tasks are highly subjective and dependent on
human factors, such as one's environment, attitudes, beliefs, and lived
experiences. Thus, employing LLMs (which do not have such human factors) in
these tasks may result in a lack of variation in data, failing to reflect the
diversity of human experiences. In this paper, we examine the role of prompting
LLMs with human-like personas and asking the models to answer as if they were a
specific human. This is done explicitly, with exact demographics, political
beliefs, and lived experiences, or implicitly via names prevalent in specific
populations. The LLM personas are then evaluated via (1) subjective annotation
task (e.g., detecting toxicity) and (2) a belief generation task, where both
tasks are known to vary across human factors. We examine the impact of explicit
vs. implicit personas and investigate which human factors LLMs recognize and
respond to. Results show that LLM personas show mixed results when reproducing
known human biases, but generate generally fail to demonstrate implicit biases.
We conclude that LLMs lack the intrinsic cognitive mechanisms of human thought,
while capturing the statistical patterns of how people speak, which may
restrict their effectiveness in complex social science applications.

通过以人类为模版的角色对大型语言模型进行提示和回答问题，我们研究了这种模型在主观注释任务和信念生成任务中的表现，结果显示大型语言模型在模拟已知的人类偏见方面存在不同的结果，但在表现隐含的偏见方面通常未能达到预期。我们得出结论，大型语言模型缺乏人类思维的内在认知机制，虽然能够捕获人们言语的统计模式，但在复杂社会科学应用中可能限制其效果。