The proliferation of automatic faithfulness metrics for summarization has
produced a need for benchmarks to evaluate them. While existing benchmarks
measure the correlation with human judgements of faithfulness on
model-generated summaries, they are insufficient for diagnosing whether metrics
are: 1) consistent, i.e., decrease as errors are introduced into a summary, 2)
effective on human-written texts, and 3) sensitive to different error types (as
summaries can contain multiple errors). To address these needs, we present a
benchmark of unfaithful minimal pairs (BUMP), a dataset of 889 human-written,
minimally different summary pairs, where a single error (from an ontology of 7
types) is introduced to a summary from the CNN/DailyMail dataset to produce an
unfaithful summary. We find BUMP complements existing benchmarks in a number of
ways: 1) the summaries in BUMP are harder to discriminate and less probable
under SOTA summarization models, 2) BUMP enables measuring the consistency of
metrics, and reveals that the most discriminative metrics tend not to be the
most consistent, 3) BUMP enables the measurement of metrics' performance on
individual error types and highlights areas of weakness for future work.

本研究提出了不忠实最小分组（BUMP）的基准测试，该数据集由 889 个人类编写的最小不同摘要对组成，摘要来自 CNN / DailyMail 数据集，其中从 7 种类型的本体论引入单个错误以生成不忠实的摘要。该基准测试使度量稳定性的测量成为可能，揭示了最具辨别力的度量标准往往不是最稳健的，并突出了度量标准在个别错误类型上的表现和未来工作的不足之处。