In this paper, we introduce a two-level attention schema, Poolingformer, for long document modeling. Its first level uses a smaller sliding window pattern to aggregate information from neighbors. Its second level employs a larger window to increase receptive fields with pooling attention to reduce both computational cost and memory consumption. We first evaluate Poolingformer on two long sequence QA tasks: the monolingual NQ and the multilingual TyDi QA. Experimental results show that Poolingformer sits atop three official leaderboards measured by F1, outperforming previous state-of-the-art models by 1.9 points (79.8 vs. 77.9) on NQ long answer, 1.9 points (79.5 vs. 77.6) on TyDi QA passage answer, and 1.6 points (67.6 vs. 66.0) on TyDi QA minimal answer. We further evaluate Poolingformer on a long sequence summarization task. Experimental results on the arXiv benchmark continue to demonstrate its superior performance.

本文介绍了一种二级注意力模式，Poolingformer，用于长文档建模，其第一级使用较小的滑动窗口模式来聚合周围信息，第二级使用更大的窗口来增加接受场并使用池化注意力来减少计算成本和内存消耗。实验结果表明，Poolingformer在三个基准测试中领先于现有的最先进模型，在长序列QA任务和长序列摘要任务上表现出优异性能。

Poolingformer: 带池化注意力的长文档建模