Commonly adopted metrics for extractive summarization focus on lexical
overlap at the token level. In this paper, we present a facet-aware evaluation
setup for better assessment of the information coverage in extracted summaries.
Specifically, we treat each sentence in the reference summary as a
\textit{facet}, identify the sentences in the document that express the
semantics of each facet as \textit{support sentences} of the facet, and
automatically evaluate extractive summarization methods by comparing the
indices of extracted sentences and support sentences of all the facets in the
reference summary. To facilitate this new evaluation setup, we construct an
extractive version of the CNN/Daily Mail dataset and perform a thorough
quantitative investigation, through which we demonstrate that facet-aware
evaluation manifests better correlation with human judgment than ROUGE, enables
fine-grained evaluation as well as comparative analysis, and reveals valuable
insights of state-of-the-art summarization methods. Data can be found at
this https URL

本文旨在为提高文本摘要信息覆盖性的评估提供一个新的基于 Facet 的评估方式，通过构建 CNN / Daily Mail 数据集的提取版本，在量化研究中证明，相对于 ROUGE，facet-aware 评估更好地与人类判断相关，能够实现细粒度评估和比较分析，并揭示了最先进的摘要方法的有价值的见解。