The advanced role-playing capabilities of Large Language Models (LLMs) have paved the way for developing Role-Playing Agents (RPAs). However, existing benchmarks, such as HPD, which incorporates manually scored character relationships into the context for LLMs to sort coherence, and SocialBench, which uses specific profiles generated by LLMs in the context of multiple-choice tasks to assess character preferences, face limitations like poor generalizability, implicit and inaccurate judgments, and excessive context length. To address the above issues, we propose an automatic, scalable, and generalizable paradigm. Specifically, we construct a benchmark by extracting relations from a general knowledge graph and leverage RPA's inherent hallucination properties to prompt it to interact across roles, employing ChatGPT for stance detection and defining relationship hallucination along with three related metrics. Extensive experiments validate the effectiveness and stability of our metrics. Our findings further explore factors influencing these metrics and discuss the trade-off between relationship hallucination and factuality.

本研究解决了现有角色扮演代理（RPA）评估中普遍存在的广泛适应性差和判断不准确等问题。我们提出了一种自动化、可扩展且具有通用性的评估范式，通过使用一般知识图谱提取关系，并利用RPA的幻想特性，进行角色间的互动评估。实验结果显示，我们的方法有效且可靠，为角色关系的评估提供了新的见解。

从一般到具体：利用一般幻想自动测量特定角色扮演代理的角色关系忠实度