The problems of unfaithful summaries have been widely discussed under the
context of abstractive summarization. Though extractive summarization is less
prone to the common unfaithfulness issues of abstractive summaries, does that
mean extractive is equal to faithful? Turns out that the answer is no. In this
work, we define a typology with five types of broad unfaithfulness problems
(including and beyond not-entailment) that can appear in extractive summaries,
including incorrect coreference, incomplete coreference, incorrect discourse,
incomplete discourse, as well as other misleading information. We ask humans to
label these problems out of 1500 English summaries produced by 15 diverse
extractive systems. We find that 33% of the summaries have at least one of the
five issues. To automatically detect these problems, we find that 5 existing
faithfulness evaluation metrics for summarization have poor correlations with
human judgment. To remedy this, we propose a new metric, ExtEval, that is
designed for detecting unfaithful extractive summaries and is shown to have the
best performance. We hope our work can increase the awareness of unfaithfulness
problems in extractive summarization and help future work to evaluate and
resolve these issues. Our data and code are publicly available at
this https URL

本论文通过定义五个宽泛的不忠实问题类型（包括和超越非蕴涵）以及其他误导信息，论证了导致摘要不忠实的核心问题主要出现在 extractive summarization 中的指代和话语上，提出了新的检测误差的方法 ExtEval，并指出现存的五大检测方法和人类判断较差，调查结果显示 15 个多样的提取系统中，33% 的摘要存在至少一个不忠实问题，旨在提高外部总结的还原度并解决其问题