The advancement of Large Language Models (LLMs) has led to significant
enhancements in the performance of chatbot systems. Many researchers have
dedicated their efforts to the development of bringing characteristics to
chatbots. While there have been commercial products for developing role-driven
chatbots using LLMs, it is worth noting that academic research in this area
remains relatively scarce. Our research focuses on investigating the
performance of LLMs in constructing Characteristic AI Agents by simulating
real-life individuals across different settings. Current investigations have
primarily focused on act on roles with simple profiles. In response to this
research gap, we create a benchmark for the characteristic AI agents task,
including dataset, techniques, and evaluation metrics. A dataset called
``Character100'' is built for this benchmark, comprising the most-visited
people on Wikipedia for language models to role-play. With the constructed
dataset, we conduct comprehensive assessment of LLMs across various settings.
In addition, we devise a set of automatic metrics for quantitative performance
evaluation. The experimental results underscore the potential directions for
further improvement in the capabilities of LLMs in constructing characteristic
AI agents. The benchmark is available at
this https URL

大语言模型在构建特色人工智能代理方面的性能评估以及自动度量指标的定量评估，突显了进一步提高大语言模型在构建特色人工智能代理方面能力的潜在方向。