The quality of a summarization evaluation metric is quantified by calculating
the correlation between its scores and human annotations across a large number
of summaries. Currently, it is unclear how precise these correlation estimates
are, nor whether differences between two metrics' correlations reflect a true
difference or if it is due to mere chance. In this work, we address these two
problems by proposing methods for calculating confidence intervals and running
hypothesis tests for correlations using two resampling methods, bootstrapping
and permutation. After evaluating which of the proposed methods is most
appropriate for summarization through two simulation experiments, we analyze
the results of applying these methods to several different automatic evaluation
metrics across three sets of human annotations. We find that the confidence
intervals are rather wide, demonstrating high uncertainty in the reliability of
automatic metrics. Further, although many metrics fail to show statistical
improvements over ROUGE, two recent works, QAEval and BERTScore, do in some
evaluation settings.

本文介绍了利用自主重采样方法来计算置信区间和运行假设检验的方法， 并对几个自动评估度量进行了分析，发现自动评估度量的可靠性存在高度不确定性