As Large Language Models (LLMs) become widely used to model and simulate
human behavior, understanding their biases becomes critical. We developed an
experimental framework using Big Five personality surveys and uncovered a
previously undetected social desirability bias in a wide range of LLMs. By
systematically varying the number of questions LLMs were exposed to, we
demonstrate their ability to infer when they are being evaluated. When
personality evaluation is inferred, LLMs skew their scores towards the
desirable ends of trait dimensions (i.e., increased extraversion, decreased
neuroticism, etc). This bias exists in all tested models, including GPT-4/3.5,
Claude 3, Llama 3, and PaLM-2. Bias levels appear to increase in more recent
models, with GPT-4's survey responses changing by 1.20 (human) standard
deviations and Llama 3's by 0.98 standard deviations-very large effects. This
bias is robust to randomization of question order and paraphrasing.
Reverse-coding all the questions decreases bias levels but does not eliminate
them, suggesting that this effect cannot be attributed to acquiescence bias.
Our findings reveal an emergent social desirability bias and suggest
constraints on profiling LLMs with psychometric tests and on using LLMs as
proxies for human participants.

通过使用大五人格调查，我们在各种大型语言模型中发现了一种以前未被发现的社会期望偏差，它会影响模型的评估和得分。这种偏差存在于所有测试的模型中，并且可能在更近期的模型中出现更严重的程度，对于使用心理测验或将模型作为人类参与者的代理存在一定的限制。