Large language models (LLMs) are now increasingly utilized for role-playing
tasks, especially in impersonating domain-specific experts, primarily through
role-playing prompts. When interacting in real-world scenarios, the
decision-making abilities of a role significantly shape its behavioral
patterns. In this paper, we concentrate on evaluating the decision-making
abilities of LLMs post role-playing thereby validating the efficacy of
role-playing. Our goal is to provide metrics and guidance for enhancing the
decision-making abilities of LLMs in role-playing tasks. Specifically, we first
use LLMs to generate virtual role descriptions corresponding to the 16
personality types of Myers-Briggs Type Indicator (abbreviated as MBTI)
representing a segmentation of the population. Then we design specific
quantitative operations to evaluate the decision-making abilities of LLMs post
role-playing from four aspects: adaptability, exploration$\&$exploitation
trade-off ability, reasoning ability, and safety. Finally, we analyze the
association between the performance of decision-making and the corresponding
MBTI types through GPT-4. Extensive experiments demonstrate stable differences
in the four aspects of decision-making abilities across distinct roles,
signifying a robust correlation between decision-making abilities and the roles
emulated by LLMs. These results underscore that LLMs can effectively
impersonate varied roles while embodying their genuine sociological
characteristics.

通过评估大型语言模型在角色扮演后的决策能力，验证了角色扮演的有效性，提供了提升大型语言模型在角色扮演任务中决策能力的度量和指导。通过使用大型语言模型生成与 MBTI 人格类型相对应的虚拟角色描述，并设计具体的量化操作来评估大型语言模型在角色扮演后的决策能力，从适应性、探索与利用权衡能力、推理能力和安全性四个方面分析决策能力与相应 MBTI 类型的关联。实验结果表明，不同角色在决策能力的四个方面存在稳定的差异，显示出大型语言模型可以有效地扮演不同角色并体现其真实的社会学特征。

关于使用大型语言模型进行角色扮演中的决策能力

On the Decision-Making Abilities in Role-Playing using Large Language  Models

We conduct a pilot study selectively evaluating the cognitive abilities
(decision making and spatial reasoning) of two recently released generative
transformer models, ChatGPT and DALL-E 2. Input prompts were constructed
following neutral a priori guidelines, rather than adversarial intent. Post hoc
qualitative analysis of the outputs shows that DALL-E 2 is able to generate at
least one correct image for each spatial reasoning prompt, but most images
generated are incorrect (even though the model seems to have a clear
understanding of the objects mentioned in the prompt). Similarly, in evaluating
ChatGPT on the rationality axioms developed under the classical Von
Neumann-Morgenstern utility theorem, we find that, although it demonstrates
some level of rational decision-making, many of its decisions violate at least
one of the axioms even under reasonable constructions of preferences, bets, and
decision-making prompts. ChatGPT's outputs on such problems generally tended to
be unpredictable: even as it made irrational decisions (or employed an
incorrect reasoning process) for some simpler decision-making problems, it was
able to draw correct conclusions for more complex bet structures. We briefly
comment on the nuances and challenges involved in scaling up such a 'cognitive'
evaluation or conducting it with a closed set of answer keys ('ground truth'),
given that these models are inherently generative and open-ended in responding
to prompts.

本篇研究通过对两个生成式变压器模型 ChatGPT 和 DALL-E 2 进行认知能力的选择性评估，发现 DALL-E 2 在空间推理方面有较好表现，而 ChatGPT 在合理构建下呈现出一定的合理性但也存在不合理决策，在模型认知能力评估规模扩大及答案具备的挑战方面存在困难。