The evaluation of abstractive summarization models typically uses test data
that is identically distributed as training data. In real-world practice,
documents to be summarized may contain input noise caused by text extraction
artifacts or data pipeline bugs. The robustness of model performance under
distribution shift caused by such noise is relatively under-studied. We present
a large empirical study quantifying the sometimes severe loss in performance
(up to 12 ROUGE-1 points) from different types of input noise for a range of
datasets and model sizes. We then propose a light-weight method for detecting
and removing such noise in the input during model inference without requiring
any extra training, auxiliary models, or even prior knowledge of the type of
noise. Our proposed approach effectively mitigates the loss in performance,
recovering a large fraction of the performance drop, sometimes as large as 11
ROUGE-1 points.

本文介绍了一个大型实证研究，量化了不同类型噪声的严重性损失，以及提出的一种轻量级方法，用于检测和消除模型推理中的输入噪声，有效地缓解了性能降低。