By harnessing pre-trained language models, summarization models had rapid
progress recently. However, the models are mainly assessed by automatic
evaluation metrics such as ROUGE. Although ROUGE is known for having a positive
correlation with human evaluation scores, it has been criticized for its
vulnerability and the gap between actual qualities. In this paper, we compare
the generated summaries from recent LM, BART, and the reference summaries from
a benchmark dataset, CNN/DM, using a crowd-sourced human evaluation metric.
Interestingly, model-generated summaries receive higher scores relative to
reference summaries. Stemming from our experimental results, we first argue the
intrinsic characteristics of the CNN/DM dataset, the progress of pre-trained
language models, and their ability to generalize on the training data. Finally,
we share our insights into the model-generated summaries and presents our
thought on learning methods for abstractive summarization.

本文探讨基于预训练语言模型的摘要生成模型。通过与基准数据集 CNN/DM 的参考摘要的人工评估比较，发现相对于参考摘要而言，由最新的语言模型 BART 生成更高分的摘要。我们对 CNN/DM 数据集内在特性、预训练语言模型的进展及其对训练数据的泛化能力进行了分析，最终提出了对于提高抽象化摘要生成的学习方法的思考。