The recent success of large language models (LLMs) has attracted widespread
interest to develop role-playing conversational agents personalized to the
characteristics and styles of different speakers to enhance their abilities to
perform both general and special purpose dialogue tasks. However, the ability
to personalize the generated utterances to speakers, whether conducted by human
or LLM, has not been well studied. To bridge this gap, our study introduces a
novel evaluation challenge: speaker verification in agent-generated
conversations, which aimed to verify whether two sets of utterances originate
from the same speaker. To this end, we assemble a large dataset collection
encompassing thousands of speakers and their utterances. We also develop and
evaluate speaker verification models under experiment setups. We further
utilize the speaker verification models to evaluate the personalization
abilities of LLM-based role-playing models. Comprehensive experiments suggest
that the current role-playing models fail in accurately mimicking speakers,
primarily due to their inherent linguistic characteristics.

最近大型语言模型的成功引起了广泛关注，发展适应不同说话者特征和风格的角色扮演对话代理人以增强其执行一般和特殊目的对话任务的能力，然而，个性化生成话语的能力，无论是由人类还是大型语言模型进行，尚未得到很好的研究。为了填补这一差距，我们的研究引入了一个新颖的评估挑战：在代理人生成的对话中进行说话人验证，目的是验证两组话语是否来自同一个说话人。为此，我们收集了一个包含数千名发言人及其话语的大型数据集，并开展了实验设置下的说话人验证模型的开发和评估。我们进一步利用说话人验证模型评估了基于大型语言模型的角色扮演模型的个性化能力。全面的实验表明，当前的角色扮演模型未能准确模仿说话者，主要原因是它们固有的语言特点。