Transformer-based models are unable to process long sequences due to their
self-attention operation, which scales quadratically with the sequence length.
To address this limitation, we introduce the Longformer with an attention
mechanism that scales linearly with sequence length, making it easy to process
documents of thousands of tokens or longer. Longformer's attention mechanism is
a drop-in replacement for the standard self-attention and combines a local
windowed attention with a task motivated global attention. Following prior work
on long-sequence transformers, we evaluate Longformer on character-level
language modeling and achieve state-of-the-art results on text8 and enwik8. In
contrast to most prior work, we also pretrain Longformer and finetune it on a
variety of downstream tasks. Our pretrained Longformer consistently outperforms
RoBERTa on long document tasks and sets new state-of-the-art results on WikiHop
and TriviaQA. We finally introduce the Longformer-Encoder-Decoder (LED), a
Longformer variant for supporting long document generative sequence-to-sequence
tasks, and demonstrate its effectiveness on the arXiv summarization dataset.

Longformer 是一种基于 Transformers 的模型，采用能够线性缩放序列长度的自注意力机制，使得处理数千个记号以上的文档变得容易；与先前的一些工作不同的是，Longformer 同时进行了预训练和下游任务的微调，并在 WikiHop 和 TriviaQA 等任务上取得了新的最优结果。此外，Longformer 还引入了 Longformer-Encoder-Decoder（LED）以支持长文档的生成序列对序列任务。