As Large Language Models (LLMs) are integrated with human daily applications
rapidly, many societal and ethical concerns are raised regarding the behavior
of LLMs. One of the ways to comprehend LLMs' behavior is to analyze their
personalities. Many recent studies quantify LLMs' personalities using
self-assessment tests that are created for humans. Yet many critiques question
the applicability and reliability of these self-assessment tests when applied
to LLMs. In this paper, we investigate LLM personalities using an alternate
personality measurement method, which we refer to as the external evaluation
method, where instead of prompting LLMs with multiple-choice questions in the
Likert scale, we evaluate LLMs' personalities by analyzing their responses
toward open-ended situational questions using an external machine learning
model. We first fine-tuned a Llama2-7B model as the MBTI personality predictor
that outperforms the state-of-the-art models as the tool to analyze LLMs'
responses. Then, we prompt the LLMs with situational questions and ask them to
generate Twitter posts and comments, respectively, in order to assess their
personalities when playing two different roles. Using the external personality
evaluation method, we identify that the obtained personality types for LLMs are
significantly different when generating posts versus comments, whereas humans
show a consistent personality profile in these two different situations. This
shows that LLMs can exhibit different personalities based on different
scenarios, thus highlighting a fundamental difference between personality in
LLMs and humans. With our work, we call for a re-evaluation of personality
definition and measurement in LLMs.

通过使用外部评估方法，本文调查了大型语言模型（LLMs）的个性，并发现在不同情境下生成帖子和评论时，LLMs 的个性有显著差异，与人类表现出一致的个性特征形成了对比，从而引发对 LLMs 个性定义和测量的重新评估。

利用外部评估在大型语言模型中识别多个人格

Identifying Multiple Personalities in Large Language Models with  External Evaluation

Comprehending characters' personalities is a crucial aspect of story reading.
As readers engage with a story, their understanding of a character evolves
based on new events and information; and multiple fine-grained aspects of
personalities can be perceived. This leads to a natural problem of situated and
fine-grained personality understanding. The problem has not been studied in the
NLP field, primarily due to the lack of appropriate datasets mimicking the
process of book reading. We present the first labeled dataset PersoNet for this
problem. Our novel annotation strategy involves annotating user notes from
online reading apps as a proxy for the original books. Experiments and human
studies indicate that our dataset construction is both efficient and accurate;
and our task heavily relies on long-term context to achieve accurate
predictions for both machines and humans. The dataset is available at
this https URL

这篇论文介绍了一个名为 PersoNet 的数据集，该数据集是研究角色个性的第一个标记数据集，并且通过在线阅读应用程序中用户笔记的注释策略进行了标注。实验和人类研究表明，我们的数据集构建既高效又准确，并且我们的任务需要长期的上下文才能为机器和人类获得准确的预测结果。

阅读小说时虚构人物性格理解

Personality Understanding of Fictional Characters during Book Reading

Human language expression is based on the subjective construal of the
situation instead of the objective truth conditions, which means that speakers'
personalities and emotions after cognitive processing have an important
influence on conversation. However, most existing datasets for conversational
AI ignore human personalities and emotions, or only consider part of them. It's
difficult for dialogue systems to understand speakers' personalities and
emotions although large-scale pre-training language models have been widely
used. In order to consider both personalities and emotions in the process of
conversation generation, we propose CPED, a large-scale Chinese personalized
and emotional dialogue dataset, which consists of multi-source knowledge
related to empathy and personal characteristic. These knowledge covers gender,
Big Five personality traits, 13 emotions, 19 dialogue acts and 10 scenes. CPED
contains more than 12K dialogues of 392 speakers from 40 TV shows. We release
the textual dataset with audio features and video features according to the
copyright claims, privacy issues, terms of service of video platforms. We
provide detailed description of the CPED construction process and introduce
three tasks for conversational AI, including personality recognition, emotion
recognition in conversations as well as personalized and emotional conversation
generation. Finally, we provide baseline systems for these tasks and consider
the function of speakers' personalities and emotions on conversation. Our
motivation is to propose a dataset to be widely adopted by the NLP community as
a new open benchmark for conversational AI research. The full dataset is
available at this https URL.

研究表明，说话者的个性和情感对对话有重要影响，因此考虑个性和情感的影响对于对话生成非常重要。为了解决这个问题，我们提出了 CPED: 一个大型、个性化和情感化的对话数据集，包含了与移情和个性特征相关的多源知识，涵盖性别、大五人格特质、13 种情感、19 种对话行为和 10 个场景。该数据集包含了来自 40 个电视节目的 392 个说话者的 12,000 多个对话，是新一代对话人工智能研究的新开放基准。