A typical journalistic convention in news articles is to deliver the most
salient information in the beginning, also known as the lead bias. While this
phenomenon can be exploited in generating a summary, it has a detrimental
effect on teaching a model to discriminate and extract important information in
general. We propose that this lead bias can be leveraged in our favor in a
simple and effective way to pre-train abstractive news summarization models on
large-scale unlabeled news corpora: predicting the leading sentences using the
rest of an article. We collect a massive news corpus and conduct data cleaning
and filtering via statistical analysis. We then apply self-supervised
pre-training on this dataset to existing generation models BART and T5 for
domain adaptation. Via extensive experiments on six benchmark datasets, we show
that this approach can dramatically improve the summarization quality and
achieve state-of-the-art results for zero-shot news summarization without any
fine-tuning. For example, in the DUC2003 dataset, the ROUGE-1 score of BART
increases 13.7% after the lead-bias pre-training. We deploy the model in
Microsoft News and provide public APIs as well as a demo website for
multi-lingual news summarization.

本文提出了一种利用新闻报道中引文偏见进行先训练的方法，以预测文章中的前导句。作者把这种方法应用到 BART 和 T5 模型中，为领域适应打下基础，并通过六个基准数据集的实验表明，这种方法可以显著提高摘要质量，取得了零调整的新闻摘要的最新成果。