Transformer-based models have achieved state-of-the-art results in a wide
range of natural language processing (NLP) tasks including document
summarization. Typically these systems are trained by fine-tuning a large
pre-trained model to the target task. One issue with these transformer-based
models is that they do not scale well in terms of memory and compute
requirements as the input length grows. Thus, for long document summarization,
it can be challenging to train or fine-tune these models. In this work, we
exploit large pre-trained transformer-based models and address long-span
dependencies in abstractive summarization using two methods: local
self-attention; and explicit content selection. These approaches are compared
on a range of network configurations. Experiments are carried out on standard
long-span summarization tasks, including Spotify Podcast, arXiv, and PubMed
datasets. We demonstrate that by combining these methods, we can achieve
state-of-the-art results on all three tasks in the ROUGE scores. Moreover,
without a large-scale GPU card, our approach can achieve comparable or better
results than existing approaches.

本研究提出使用 local self-attention 和 explicit content selection 两种方法来应对长篇文档摘要中的长跨度依赖，并利用大型预训练变压器模型在 Spotify Podcast、arXiv 和 PubMed 等标准数据集上进行实验，证明两种方法的组合可以在 3 个任务中实现 ROUGE 得分的最优结果。此外，与现有算法相比，在没有大规模 GPU 计算机卡的情况下，本研究的方法也可以实现相当或更好的结果。