In text summarization, evaluating the efficacy of automatic metrics without
human judgments has become recently popular. One exemplar work concludes that
automatic metrics strongly disagree when ranking high-scoring summaries. In
this paper, we revisit their experiments and find that their observations stem
from the fact that metrics disagree in ranking summaries from any narrow
scoring range. We hypothesize that this may be because summaries are similar to
each other in a narrow scoring range and are thus, difficult to rank. Apart
from the width of the scoring range of summaries, we analyze three other
properties that impact inter-metric agreement - Ease of Summarization,
Abstractiveness, and Coverage. To encourage reproducible research, we make all
our analysis code and data publicly available.

本文讨论了人工评价与自动评价在文本摘要中的应用，并提出了评价指标间的差异和相关性。研究发现评价指标的互相协议性取决于摘要的得分范围、摘要易于摘要的程度、摘要抽象程度和覆盖范围等方面。