Evaluating multi-document summarization (MDS) quality is difficult. This is
especially true in the case of MDS for biomedical literature reviews, where
models must synthesize contradicting evidence reported across different
documents. Prior work has shown that rather than performing the task, models
may exploit shortcuts that are difficult to detect using standard n-gram
similarity metrics such as ROUGE. Better automated evaluation metrics are
needed, but few resources exist to assess metrics when they are proposed.
Therefore, we introduce a dataset of human-assessed summary quality facets and
pairwise preferences to encourage and support the development of better
automated evaluation methods for literature review MDS. We take advantage of
community submissions to the Multi-document Summarization for Literature Review
(MSLR) shared task to compile a diverse and representative sample of generated
summaries. We analyze how automated summarization evaluation metrics correlate
with lexical features of generated summaries, to other automated metrics
including several we propose in this work, and to aspects of human-assessed
summary quality. We find that not only do automated metrics fail to capture
aspects of quality as assessed by humans, in many cases the system rankings
produced by these metrics are anti-correlated with rankings according to human
annotators.

为了改进文献综述的多文件摘要 (MDS)，引入包含人工评估的摘要质量特征和成对偏好的数据集，以支持开发更好的自动评估方法，分析了自动摘要评估度量与生成的摘要词汇特征、其他自动度量以及人工评估摘要质量方面的相关性。发现自动化度量无法完全捕捉到人工评估所评估的质量方面。