As Large Language Models (LLMs) are integrated with human daily applications
rapidly, many societal and ethical concerns are raised regarding the behavior
of LLMs. One of the ways to comprehend LLMs' behavior is to analyze their
personalities. Many recent studies quantify LLMs' personalities using
self-assessment tests that are created for humans. Yet many critiques question
the applicability and reliability of these self-assessment tests when applied
to LLMs. In this paper, we investigate LLM personalities using an alternate
personality measurement method, which we refer to as the external evaluation
method, where instead of prompting LLMs with multiple-choice questions in the
Likert scale, we evaluate LLMs' personalities by analyzing their responses
toward open-ended situational questions using an external machine learning
model. We first fine-tuned a Llama2-7B model as the MBTI personality predictor
that outperforms the state-of-the-art models as the tool to analyze LLMs'
responses. Then, we prompt the LLMs with situational questions and ask them to
generate Twitter posts and comments, respectively, in order to assess their
personalities when playing two different roles. Using the external personality
evaluation method, we identify that the obtained personality types for LLMs are
significantly different when generating posts versus comments, whereas humans
show a consistent personality profile in these two different situations. This
shows that LLMs can exhibit different personalities based on different
scenarios, thus highlighting a fundamental difference between personality in
LLMs and humans. With our work, we call for a re-evaluation of personality
definition and measurement in LLMs.

通过使用外部评估方法，本文调查了大型语言模型（LLMs）的个性，并发现在不同情境下生成帖子和评论时，LLMs 的个性有显著差异，与人类表现出一致的个性特征形成了对比，从而引发对 LLMs 个性定义和测量的重新评估。

利用外部评估在大型语言模型中识别多个人格

Identifying Multiple Personalities in Large Language Models with  External Evaluation

As large language models (LLM) evolve in their capabilities, various recent
studies have tried to quantify their behavior using psychological tools created
to study human behavior. One such example is the measurement of "personality"
of LLMs using personality self-assessment tests. In this paper, we take three
such studies on personality measurement of LLMs that use personality
self-assessment tests created to study human behavior. We use the prompts used
in these three different papers to measure the personality of the same LLM. We
find that all three prompts lead very different personality scores. This simple
test reveals that personality self-assessment scores in LLMs depend on the
subjective choice of the prompter. Since we don't know the ground truth value
of personality scores for LLMs as there is no correct answer to such questions,
there's no way of claiming if one prompt is more or less correct than the
other. We then introduce the property of option order symmetry for personality
measurement of LLMs. Since most of the self-assessment tests exist in the form
of multiple choice question (MCQ) questions, we argue that the scores should
also be robust to not just the prompt template but also the order in which the
options are presented. This test unsurprisingly reveals that the answers to the
self-assessment tests are not robust to the order of the options. These simple
tests, done on ChatGPT and Llama2 models show that self-assessment personality
tests created for humans are not appropriate for measuring personality in LLMs.

大规模语言模型的人格测量结果依赖于自我评估测试的选择，而人类使用的测试无法准确衡量 LLM 的人格特点。