Text summarization has a wide range of applications in many scenarios. The
evaluation of the quality of the generated text is a complex problem. A big
challenge to language evaluation is that there is a clear divergence between
existing metrics and human evaluation. For example, the quality of a document
summary can be measured by human annotators from both objective aspects, such
as grammatical and semantic correctness, as well as subjective dimensions, such
as comprehensiveness, succinctness, and interestingness. Most of the automatic
evaluation methods like BLUE/ROUGE may be not able to capture the above
dimensions well. In this paper, we propose a new evaluation framework based on
LLMs, which provides a comprehensive evaluation framework by comparing
generated text and reference text from both objective and subjective aspects.
First, we propose to model objective and subjective dimensions of generated
text based on roleplayers prompting mechanism. Furthermore, we introduce a
context-based prompting mechanism that is able to generate dynamic roleplayer
profiles based on input context. Finally, we design a multi-roleplayer
prompting technology based on batch prompting to integrate multiple evaluation
results into evaluation results. Experimental results on two real datasets for
summarization show that our model is highly competitive and has a very high
consistency with human annotators.

本文提出了一种新的评估框架，基于 LLMs，并通过比较生成文本和参考文本来提供全面的评估。该模型基于角色扮演者提示机制模拟生成文本的客观和主观维度，并引入了上下文提示机制以生成基于输入上下文的动态角色扮演者配置文件，并根据批处理提示设计了多角色扮演者提示技术，以将多个评估结果集成到评估结果中。在自动摘要任务的两个真实数据集上进行的实验结果表明，该模型非常具有竞争力，且与人类注释者具有非常高的一致性。