How reliably an automatic summarization evaluation metric replicates human
judgments of summary quality is quantified by system-level correlations. We
identify two ways in which the definition of the system-level correlation is
inconsistent with how metrics are used to evaluate systems in practice and
propose changes to rectify this disconnect. First, we calculate the system
score for an automatic metric using the full test set instead of the subset of
summaries judged by humans, which is currently standard practice. We
demonstrate how this small change leads to more precise estimates of
system-level correlations. Second, we propose to calculate correlations only on
pairs of systems that are separated by small differences in automatic scores
which are commonly observed in practice. This allows us to demonstrate that our
best estimate of the correlation of ROUGE to human judgments is near 0 in
realistic scenarios. The results from the analyses point to the need to collect
more high-quality human judgments and to improve automatic metrics when
differences in system scores are small.

本文探讨了自动文摘评估指标能够准确复制人类总结质量判断的可靠性，并提出了对现有评估体系的变更，以改善两个方面的偏差，这两个方面分别是评估指标使用的方法与实际评估系统的方式不一致，以及通过只计算小得分差距的系统之间的相关性来计算相关性。通过此研究，提出了收集更多高质量的人类判断和在系统得分差距小的情况下改进自动评估指标的必要性。