Vision Transformers achieved outstanding performance in many computer vision
tasks. Early Vision Transformers such as ViT and DeiT adopt global
self-attention, which is computationally expensive when the number of patches
is large. To improve efficiency, recent Vision Transformers adopt local
self-attention mechanisms, where self-attention is computed within local
windows. Despite the fact that window-based local self-attention significantly
boosts efficiency, it fails to capture the relationships between distant but
similar patches in the image plane. To overcome this limitation of image-space
local attention, in this paper, we further exploit the locality of patches in
the feature space. We group the patches into multiple clusters using their
features, and self-attention is computed within every cluster. Such
feature-space local attention effectively captures the connections between
patches across different local windows but still relevant. We propose a
Bilateral lOcal Attention vision Transformer (BOAT), which integrates
feature-space local attention with image-space local attention. We further
integrate BOAT with both Swin and CSWin models, and extensive experiments on
several benchmark datasets demonstrate that our BOAT-CSWin model clearly and
consistently outperforms existing state-of-the-art CNN models and vision
Transformers.

本文提出了一种新的局部自注意机制 BOAT，该机制在特征空间中对局部窗口中的补丁进行聚类，并在每个聚类中进行自我注意。将 BOAT 集成进 CSWin 模型，实验结果表明，BOAT-CSWin 模型在多个基准数据集上表现良好，明显优于现有的卷积神经网络模型和视觉 Transformer。

BOAT: 双向局部注意力视觉 Transformer

BOAT: Bilateral Local Attention Vision Transformer

Self-attention is powerful in modeling long-range dependencies, but it is
weak in local finer-level feature learning. The performance of local
self-attention (LSA) is just on par with convolution and inferior to dynamic
filters, which puzzles researchers on whether to use LSA or its counterparts,
which one is better, and what makes LSA mediocre. To clarify these, we
comprehensively investigate LSA and its counterparts from two sides:
\emph{channel setting} and \emph{spatial processing}. We find that the devil
lies in the generation and application of spatial attention, where relative
position embeddings and the neighboring filter application are key factors.
Based on these findings, we propose the enhanced local self-attention (ELSA)
with Hadamard attention and the ghost head. Hadamard attention introduces the
Hadamard product to efficiently generate attention in the neighboring case,
while maintaining the high-order mapping. The ghost head combines attention
maps with static matrices to increase channel capacity. Experiments demonstrate
the effectiveness of ELSA. Without architecture / hyperparameter modification,
drop-in replacing LSA with ELSA boosts Swin Transformer \cite{swin} by up to
+1.4 on top-1 accuracy. ELSA also consistently benefits VOLO \cite{volo} from
D1 to D5, where ELSA-VOLO-D5 achieves 87.2 on the ImageNet-1K without extra
training images. In addition, we evaluate ELSA in downstream tasks. ELSA
significantly improves the baseline by up to +1.9 box Ap / +1.3 mask Ap on the
COCO, and by up to +1.9 mIoU on the ADE20K. Code is available at
https://github.com/damo-cv/ELSA.

本文通过从通道设置和空间处理两个方面综合调查了本地自注意力和其同行的性能，发现空间注意力的生成和应用是影响性能的关键因素，提出了增强型本地自注意力（ELSA）并通过实验验证其有效性。

ELSA：增强的局部自注意力视觉变换器

ELSA: Enhanced Local Self-Attention for Vision Transformer

Transformer-based models have achieved state-of-the-art results in a wide
range of natural language processing (NLP) tasks including document
summarization. Typically these systems are trained by fine-tuning a large
pre-trained model to the target task. One issue with these transformer-based
models is that they do not scale well in terms of memory and compute
requirements as the input length grows. Thus, for long document summarization,
it can be challenging to train or fine-tune these models. In this work, we
exploit large pre-trained transformer-based models and address long-span
dependencies in abstractive summarization using two methods: local
self-attention; and explicit content selection. These approaches are compared
on a range of network configurations. Experiments are carried out on standard
long-span summarization tasks, including Spotify Podcast, arXiv, and PubMed
datasets. We demonstrate that by combining these methods, we can achieve
state-of-the-art results on all three tasks in the ROUGE scores. Moreover,
without a large-scale GPU card, our approach can achieve comparable or better
results than existing approaches.

本研究提出使用 local self-attention 和 explicit content selection 两种方法来应对长篇文档摘要中的长跨度依赖，并利用大型预训练变压器模型在 Spotify Podcast、arXiv 和 PubMed 等标准数据集上进行实验，证明两种方法的组合可以在 3 个任务中实现 ROUGE 得分的最优结果。此外，与现有算法相比，在没有大规模 GPU 计算机卡的情况下，本研究的方法也可以实现相当或更好的结果。

局部关注和内容选择实现的长篇摘要

Long-Span Summarization via Local Attention and Content Selection

Neural networks, particularly Transformer-based architectures, have achieved
significant performance improvements on several retrieval benchmarks. When the
items being retrieved are documents, the time and memory cost of employing
Transformers over a full sequence of document terms can be prohibitive. A
popular strategy involves considering only the first n terms of the document.
This can, however, result in a biased system that under retrieves longer
documents. In this work, we propose a local self-attention which considers a
moving window over the document terms and for each term attends only to other
terms in the same window. This local attention incurs a fraction of the compute
and memory cost of attention over the whole document. The windowed approach
also leads to more compact packing of padded documents in minibatches resulting
in additional savings. We also employ a learned saturation function and a
two-staged pooling strategy to identify relevant regions of the document. The
Transformer-Kernel pooling model with these changes can efficiently elicit
relevance information from documents with thousands of tokens. We benchmark our
proposed modifications on the document ranking task from the TREC 2019 Deep
Learning track and observe significant improvements in retrieval quality as
well as increased retrieval of longer documents at moderate increase in compute
and memory costs.

本文提出了一种局部自注意力机制，用于解决检索过程中考虑前 n 个单词导致系统检索长文档时出现偏差的问题，并在 TREC 2019 深度学习排行榜任务上取得了显著的检索质量提升。