Transformer is important for text modeling. However, it has difficulty in
handling long documents due to the quadratic complexity with input text length.
In order to handle this problem, we propose a hierarchical interactive
Transformer (Hi-Transformer) for efficient and effective long document
modeling. Hi-Transformer models documents in a hierarchical way, i.e., first
learns sentence representations and then learns document representations. It
can effectively reduce the complexity and meanwhile capture global document
context in the modeling of each sentence. More specifically, we first use a
sentence Transformer to learn the representations of each sentence. Then we use
a document Transformer to model the global document context from these sentence
representations. Next, we use another sentence Transformer to enhance sentence
modeling using the global document context. Finally, we use hierarchical
pooling method to obtain document embedding. Extensive experiments on three
benchmark datasets validate the efficiency and effectiveness of Hi-Transformer
in long document modeling.

提出一种称为 Hi-Transformer 的层次交互 Transformer 模型，通过学习每个句子的表示来获取全局文档信息，可以高效、有效地处理长文档建模问题。

Hi-Transformer: 高效、有效的长文档建模的分层交互式 Transformer

Hi-Transformer: Hierarchical Interactive Transformer for Efficient and  Effective Long Document Modeling

In this paper, we introduce a two-level attention schema, Poolingformer, for
long document modeling. Its first level uses a smaller sliding window pattern
to aggregate information from neighbors. Its second level employs a larger
window to increase receptive fields with pooling attention to reduce both
computational cost and memory consumption. We first evaluate Poolingformer on
two long sequence QA tasks: the monolingual NQ and the multilingual TyDi QA.
Experimental results show that Poolingformer sits atop three official
leaderboards measured by F1, outperforming previous state-of-the-art models by
1.9 points (79.8 vs. 77.9) on NQ long answer, 1.9 points (79.5 vs. 77.6) on
TyDi QA passage answer, and 1.6 points (67.6 vs. 66.0) on TyDi QA minimal
answer. We further evaluate Poolingformer on a long sequence summarization
task. Experimental results on the arXiv benchmark continue to demonstrate its
superior performance.

本文介绍了一种二级注意力模式，Poolingformer，用于长文档建模，其第一级使用较小的滑动窗口模式来聚合周围信息，第二级使用更大的窗口来增加接受场并使用池化注意力来减少计算成本和内存消耗。实验结果表明，Poolingformer 在三个基准测试中领先于现有的最先进模型，在长序列 QA 任务和长序列摘要任务上表现出优异性能。