Automated evaluation is crucial for streamlining text summarization
benchmarking and model development, given the costly and time-consuming nature
of human evaluation. Traditional methods like ROUGE do not correlate well with
human judgment, while recently proposed LLM-based metrics provide only
summary-level assessment using Likert-scale scores. This limits deeper model
analysis, e.g., we can only assign one hallucination score at the summary
level, while at the sentence level, we can count sentences containing
hallucinations. To remedy those limitations, we propose FineSurE, a
fine-grained evaluator specifically tailored for the summarization task using
large language models (LLMs). It also employs completeness and conciseness
criteria, in addition to faithfulness, enabling multi-dimensional assessment.
We compare various open-source and proprietary LLMs as backbones for FineSurE.
In addition, we conduct extensive benchmarking of FineSurE against SOTA methods
including NLI-, QA-, and LLM-based methods, showing improved performance
especially on the completeness and conciseness dimensions. The code is
available at this https URL

利用大型语言模型 FineSurE，我们针对文本摘要任务提出了一种细粒度评估方法，该方法在完成度、简洁度和忠实度等多个维度上对摘要性能进行评估，并在各种开源和专有的大型语言模型作为 FineSurE 的基础上进行了广泛的基准测试，改进了摘要性能。