Manual evaluation is essential to judge progress on automatic text
summarization. However, we conduct a survey on recent summarization system
papers that reveals little agreement on how to perform such evaluation studies.
We conduct two evaluation experiments on two aspects of summaries' linguistic
quality (coherence and repetitiveness) to compare Likert-type and ranking
annotations and show that best choice of evaluation method can vary from one
aspect to another. In our survey, we also find that study parameters such as
the overall number of annotators and distribution of annotators to annotation
items are often not fully reported and that subsequent statistical analysis
ignores grouping factors arising from one annotator judging multiple summaries.
Using our evaluation experiments, we show that the total number of annotators
can have a strong impact on study power and that current statistical analysis
methods can inflate type I error rates up to eight-fold. In addition, we
highlight that for the purpose of system comparison the current practice of
eliciting multiple judgements per summary leads to less powerful and reliable
annotations given a fixed study budget.

通过对最近总结系统论文的调查，我们发现，在如何进行这样的评估研究方面存在很少的一致性。我们进行了两个评估实验来比较 Likert 类型和排名注释，并展示了评估方法的最佳选择可能因一个方面与另一个方面不同。使用我们的评估实验，我们展示了注释者的总数对研究功率有很强的影响，而目前的统计分析方法在把一个注释者判定多个摘要时会夸大 Ⅰ 类错误率高达 8 倍。此外，我们强调，为了进行系统比较，当前的实践是在一个固定的研究预算中引出多个判断，这会导致注释功率和可靠性较弱。