Ideal summarization models should generalize to novel summary-worthy content
without remembering reference training summaries by rote. However, a single
average performance score on the entire test set is inadequate in determining
such model competencies. We propose a fine-grained evaluation protocol by
partitioning a test set based on the lexical similarity of reference test
summaries with training summaries. We observe up to a 5x (1.2x) difference in
ROUGE-2 (entity recall) scores between the subsets with the lowest and highest
similarity. Next, we show that such training repetitions also make a model
vulnerable to rote learning, reproducing data artifacts such as factual errors,
especially when reference test summaries are lexically close to training
summaries. Consequently, we propose to limit lexical repetitions in training
summaries during both supervised fine-tuning and likelihood calibration stages
to improve the performance on novel test cases while retaining average
performance. Our automatic and human evaluations on novel test subsets and
recent news articles show that limiting lexical repetitions in training
summaries can prevent rote learning and improve generalization.

理想的摘要模型应该能推广到新的值得摘要的内容，而不需要死记参考训练摘要，我们提出了一种细粒度的评估协议，通过基于参考测试摘要与训练摘要之间的词汇相似性将测试集划分，限制训练摘要中的词汇重复能够防止死记硬背，并提高摘要模型的泛化性。