Pretrained transformer models have demonstrated remarkable performance across
various natural language processing tasks. These models leverage the attention
mechanism to capture long- and short-range dependencies in the sequence.
However, the (full) attention mechanism incurs high computational cost -
quadratic in the sequence length, which is not affordable in tasks with long
sequences, e.g., inputs with 8k tokens. Although sparse attention can be used
to improve computational efficiency, as suggested in existing work, it has
limited modeling capacity and often fails to capture complicated dependencies
in long sequences. To tackle this challenge, we propose MASFormer, an
easy-to-implement transformer variant with Mixed Attention Spans. Specifically,
MASFormer is equipped with full attention to capture long-range dependencies,
but only at a small number of layers. For the remaining layers, MASformer only
employs sparse attention to capture short-range dependencies. Our experiments
on natural language modeling and generation tasks show that a decoder-only
MASFormer model of 1.3B parameters can achieve competitive performance to
vanilla transformers with full attention while significantly reducing
computational cost (up to 75%). Additionally, we investigate the effectiveness
of continual training with long sequence data and how sequence length impacts
downstream generation performance, which may be of independent interest.

提出了一种名为 MASFormer 的变种 Transformer 模型，它通过混合的注意力机制在处理长序列数据时既能捕捉到长距离依赖关系，又能在其余层使用稀疏注意力提高计算效率。实验结果表明，该模型在自然语言建模和生成任务中能够达到与全注意力的 Transformer 模型相媲美的性能，同时显著降低计算成本（多达 75%），并对长序列数据的持续训练和序列长度对下游生成任务的影响进行了研究。