The training of vision transformer (ViT) networks on small-scale datasets poses a significant challenge. By contrast, convolutional neural networks (CNNs) have an architectural inductive bias enabling them to perform well on such problems. In this paper, we argue that the architectural bias inherent to CNNs can be reinterpreted as an initialization bias within ViT. This insight is significant as it empowers ViTs to perform equally well on small-scale problems while maintaining their flexibility for large-scale applications. Our inspiration for this ``structured'' initialization stems from our empirical observation that random impulse filters can achieve comparable performance to learned filters within CNNs. Our approach achieves state-of-the-art performance for data-efficient ViT learning across numerous benchmarks including CIFAR-10, CIFAR-100, and SVHN.

本文探讨了视觉transformer（ViT）网络在小规模数据集上的训练问题，提出了通过重新解释卷积神经网络（CNN）的架构偏差作为ViT的初始化偏差，使其在小规模问题上表现出色，并保持其在大规模应用中的灵活性，该方法在诸多基准数据集（包括CIFAR-10、CIFAR-100和SVHN）上实现了最先进的数据高效ViT学习性能。

视觉Transformer中的结构化初始化注意力