Visual Transformers (VTs) are emerging as an architectural paradigm
alternative to Convolutional networks (CNNs). Differently from CNNs, VTs can
capture global relations between image elements and they potentially have a
larger representation capacity. However, the lack of the typical convolutional
inductive bias makes these models more data-hungry than common CNNs. In fact,
some local properties of the visual domain which are embedded in the CNN
architectural design, in VTs should be learned from samples. In this paper, we
empirically analyse different VTs, comparing their robustness in a small
training-set regime, and we show that, despite having a comparable accuracy
when trained on ImageNet, their performance on smaller datasets can be largely
different. Moreover, we propose a self-supervised task which can extract
additional information from images with only a negligible computational
overhead. This task encourages the VTs to learn spatial relations within an
image and makes the VT training much more robust when training data are scarce.
Our task is used jointly with the standard (supervised) training and it does
not depend on specific architectural choices, thus it can be easily plugged in
the existing VTs. Using an extensive evaluation with different VTs and
datasets, we show that our method can improve (sometimes dramatically) the
final accuracy of the VTs. Our code is available at:
this https URL

本文研究使用自监督任务和少量数据进行训练的 Visual Transformer 网络的表现，并发现新的自监督任务可以在空间关系方面鼓励 VT 网络，从而显著提高其小数据集准确性。