Many NLP tasks require processing long contexts beyond the length limit of
pretrained models. In order to scale these models to longer text sequences,
many efficient long-range attention variants have been proposed. Despite the
abundance of research along this direction, it is still difficult to gauge the
relative effectiveness of these models in practical use cases, e.g., if we
apply these models following the pretrain-and-finetune paradigm. In this work,
we aim to conduct a thorough analysis of these emerging models with large-scale
and controlled experiments. For each attention variant, we pretrain large-size
models using the same long-doc corpus and then finetune these models for
real-world long-context tasks. Our findings reveal pitfalls of an existing
widely-used long-range benchmark and show none of the tested efficient
attentions can beat a simple local window attention under standard pretraining
paradigms. Further analysis on local attention variants suggests that even the
commonly used attention-window overlap is not necessary to achieve good
downstream results -- using disjoint local attentions, we are able to build a
simpler and more efficient long-doc QA model that matches the performance of
Longformer~\citep{longformer} with half of its pretraining compute.
The code to replicate our experiments can be found at
this https URL

本研究通过大规模的控制实验，分析了不同的长距离注意力机制，研究发现，即使使用相同的预训练数据集与微调过程，采用简单的局部窗口注意力机制仍然能够胜任大部分任务，而且采用分离的局部注意力机制可以构建一个更为简单和高效的长文档问答模型，达到与 Longformer 半数的预训练计算相匹配的性能。