Training vision transformer networks on small datasets poses challenges. In
contrast, convolutional neural networks (CNNs) can achieve state-of-the-art
performance by leveraging their architectural inductive bias. In this paper, we
investigate whether this inductive bias can be reinterpreted as an
initialization bias within a vision transformer network. Our approach is
motivated by the finding that random impulse filters can achieve almost
comparable performance to learned filters in CNNs. We introduce a novel
initialization strategy for transformer networks that can achieve comparable
performance to CNNs on small datasets while preserving its architectural
flexibility.

我们研究了在小数据集上训练视觉转换网络所面临的挑战，发现卷积神经网络可以通过利用其体系结构归纳偏差实现最先进的性能。我们探索了这种归纳偏差是否可以重新解释为视觉转换网络中的初始化偏差，并提出了一种新的初始化策略，可以在小数据集上实现与卷积神经网络相当的性能，同时保留其架构的灵活性。

卷积初始化用于数据高效的视觉 Transformer 模型

Convolutional Initialization for Data-Efficient Vision Transformers

Self-supervised learning can be used for mitigating the greedy needs of
Vision Transformer networks for very large fully-annotated datasets. Different
classes of self-supervised learning offer representations with either good
contextual reasoning properties, e.g., using masked image modeling strategies,
or invariance to image perturbations, e.g., with contrastive methods. In this
work, we propose a single-stage and standalone method, MOCA, which unifies both
desired properties using novel mask-and-predict objectives defined with
high-level features (instead of pixel-level details). Moreover, we show how to
effectively employ both learning paradigms in a synergistic and
computation-efficient way. Doing so, we achieve new state-of-the-art results on
low-shot settings and strong experimental results in various evaluation
protocols with a training that is at least 3 times faster than prior methods.

提出了一种单阶段且独立的方法 MOCA，用高层特征定义新颖的掩码和预测目标，有效地结合了自我监督学习的两种范式，在低样本设置下取得了新的最先进结果，并且训练速度至少比之前的方法快 3 倍。