To capture the semantic graph structure from raw text, most existing
summarization approaches are built on GNNs with a pre-trained model. However,
these methods suffer from cumbersome procedures and inefficient computations
for long-text documents. To mitigate these issues, this paper proposes
HETFORMER, a Transformer-based pre-trained model with multi-granularity sparse
attentions for long-text extractive summarization. Specifically, we model
different types of semantic nodes in raw text as a potential heterogeneous
graph and directly learn heterogeneous relationships (edges) among nodes by
Transformer. Extensive experiments on both single- and multi-document
summarization tasks show that HETFORMER achieves state-of-the-art performance
in Rouge F1 while using less memory and fewer parameters.

本文提出了一种基于 Transformer 的预训练模型，通过多粒度稀疏注意力机制，从长文本中捕捉语义图结构并进行提取式摘要，实验证明 HETFORMER 在 Rouge F1 上具有最先进的性能，同时使用更少的内存和参数。