Transformer models have achieved state-of-the-art results in a wide range of
NLP tasks including summarization. Training and inference using large
transformer models can be computationally expensive. Previous work has focused
on one important bottleneck, the quadratic self-attention mechanism in the
encoder. Modified encoder architectures such as LED or LoBART use local
attention patterns to address this problem for summarization. In contrast, this
work focuses on the transformer's encoder-decoder attention mechanism. The cost
of this attention becomes more significant in inference or training approaches
that require model-generated histories. First, we examine the complexity of the
encoder-decoder attention. We demonstrate empirically that there is a sparse
sentence structure in document summarization that can be exploited by
constraining the attention mechanism to a subset of input sentences, whilst
maintaining system performance. Second, we propose a modified architecture that
selects the subset of sentences to constrain the encoder-decoder attention.
Experiments are carried out on abstractive summarization tasks, including
CNN/DailyMail, XSum, Spotify Podcast, and arXiv.

研究表明摘要任务中有稀疏的句子结构，可以用部分输入句子来限制编码器 - 解码器注意机制，并保持系统性能。