With large language models (LLMs) appearing to behave increasingly human-like
in text-based interactions, it has become popular to attempt to evaluate
various properties of these models using tests originally designed for humans.
While re-using existing tests is a resource-efficient way to evaluate LLMs,
careful adjustments are usually required to ensure that test results are even
valid across human sub-populations. Thus, it is not clear to what extent
different tests' validity generalizes to LLMs. In this work, we provide
evidence that LLMs' responses to personality tests systematically deviate from
typical human responses, implying that these results cannot be interpreted in
the same way as human test results. Concretely, reverse-coded items (e.g. "I am
introverted" vs "I am extraverted") are often both answered affirmatively by
LLMs. In addition, variation across different prompts designed to "steer" LLMs
to simulate particular personality types does not follow the clear separation
into five independent personality factors from human samples. In light of these
results, we believe it is important to pay more attention to tests' validity
for LLMs before drawing strong conclusions about potentially ill-defined
concepts like LLMs' "personality".

大型语言模型（LLMs）的人类化行为在文本交互中愈发明显，尝试使用原本为人类设计的测试来评估模型的各种特性已变得流行起来。然而，需要仔细调整这些测试，以确保测试结果在人类亚群中的有效性。因此，不清楚不同测试的有效性能够推广到 LLMs 的程度。本研究提供证据表明，LLMs 对个性测试的反应与典型人类反应存在系统偏差，意味着这些结果不能像人类测试结果一样进行解释。具体来说，LLMs 经常同时肯定地回答出了反向编码项目（例如，“我是内向的” 与 “我是外向的”）。此外，设计用于 “引导” LLMs 模拟特定的个性类型的不同提示的变化并不遵循人类样本中五个独立个性因素的明确分离。考虑到这些结果，我们认为在对类似 LLMs “个性” 这样定义模糊的概念进行强有力的结论之前，应更加关注对 LLMs 的测试有效性。