Transformers have achieved success in both language and vision domains.
However, it is prohibitively expensive to scale them to long sequences such as
long documents or high-resolution images, because self-attention mechanism has
quadratic time and memory complexities with respect to the input sequence
length. In this paper, we propose Long-Short Transformer (Transformer-LS), an
efficient self-attention mechanism for modeling long sequences with linear
complexity for both language and vision tasks. It aggregates a novel long-range
attention with dynamic projection to model distant correlations and a
short-term attention to capture fine-grained local correlations. We propose a
dual normalization strategy to account for the scale mismatch between the two
attention mechanisms. Transformer-LS can be applied to both autoregressive and
bidirectional models without additional complexity. Our method outperforms the
state-of-the-art models on multiple tasks in language and vision domains,
including the Long Range Arena benchmark, autoregressive language modeling, and
ImageNet classification. For instance, Transformer-LS achieves 0.97 test BPC on
enwik8 using half the number of parameters than previous method, while being
faster and is able to handle 3x as long sequences compared to its
full-attention version on the same hardware. On ImageNet, it can obtain the
state-of-the-art results (e.g., a moderate size of 55.8M model solely trained
on 224x224 ImageNet-1K can obtain Top-1 accuracy 84.1%), while being more
scalable on high-resolution images. The source code and models are released at
this https URL .

提出了一种名为 “长短变压器” 的模型，其中使用自注意力机制处理长文本和高分辨率图像，同时引入了一种新型的远距离关注和短期关注机制，并采用双重归一化策略来处理两种注意力机制之间的规模差异。通过在多个语言和视觉任务中的表现，该方法优于现有的方法。