The quadratic computational and memory complexities of large Transformers
have limited their scalability for long document summarization. In this paper,
we propose Hepos, a novel efficient encoder-decoder attention with head-wise
positional strides to effectively pinpoint salient information from the source.
We further conduct a systematic study of existing efficient self-attentions.
Combined with Hepos, we are able to process ten times more tokens than existing
models that use full attentions. For evaluation, we present a new dataset,
GovReport, with significantly longer documents and summaries. Results show that
our models produce significantly higher ROUGE scores than competitive
comparisons, including new state-of-the-art results on PubMed. Human evaluation
also shows that our models generate more informative summaries with fewer
unfaithful errors.

本文提出了 Hepos，一种新颖的有效的编码器 - 解码器关注机制，结合 Hepos，我们能够处理比使用全注意力的现有模型多 10 倍的令牌，有别于现有的高斯自注意机制。同时，我们还展示了新的包含非常长的文档和总结的数据集 GovReport，并且试验结果表明，我们的模型产生的 ROUGE 得分明显高于竞争对手比较，包括 PubMed 上的最新最先进的结果，人类评估也表明，我们的模型产生更多有用信息的总结。