Convolutional architectures have proven extremely successful for vision
tasks. Their hard inductive biases enable sample-efficient learning, but come
at the cost of a potentially lower performance ceiling. Vision Transformers
(ViTs) rely on more flexible self-attention layers, and have recently
outperformed CNNs for image classification. However, they require costly
pre-training on large external datasets or distillation from pre-trained
convolutional networks. In this paper, we ask the following question: is it
possible to combine the strengths of these two architectures while avoiding
their respective limitations? To this end, we introduce gated positional
self-attention (GPSA), a form of positional self-attention which can be
equipped with a ``soft" convolutional inductive bias. We initialise the GPSA
layers to mimic the locality of convolutional layers, then give each attention
head the freedom to escape locality by adjusting a gating parameter regulating
the attention paid to position versus content information. The resulting
convolutional-like ViT architecture, ConViT, outperforms the DeiT on ImageNet,
while offering a much improved sample efficiency. We further investigate the
role of locality in learning by first quantifying how it is encouraged in
vanilla self-attention layers, then analysing how it is escaped in GPSA layers.
We conclude by presenting various ablations to better understand the success of
the ConViT. Our code and models are released publicly at
this https URL

本文提出了一种形式的位置自注意力机制：门控位置自注意力，该机制能够同时具备卷积神经网络的空域学习和自注意力层的位置不变性。作者使用这种机制构建了名为 ConViT 的混合卷积 - 自注意力神经网络，通过在 ImageNet 数据集上的实验表明，该网络在图像分类任务上拥有优异的性能和更高的样本效率，并提高了对定位特征的注意力。