Vision Transformers (ViT) have been shown to attain highly competitive
performance for a wide range of vision applications, such as image
classification, object detection and semantic image segmentation. In comparison
to convolutional neural networks, the Vision Transformer's weaker inductive
bias is generally found to cause an increased reliance on model regularization
or data augmentation ("AugReg" for short) when training on smaller training
datasets. We conduct a systematic empirical study in order to better understand
the interplay between the amount of training data, AugReg, model size and
compute budget. As one result of this study we find that the combination of
increased compute and AugReg can yield models with the same performance as
models trained on an order of magnitude more training data: we train ViT models
of various sizes on the public ImageNet-21k dataset which either match or
outperform their counterparts trained on the larger, but not publicly available
JFT-300M dataset.

本文通过系统的实证研究，发现增加计算资源和数据增强可以弥补 Vision Transformers 学习小规模数据时的归纳偏差，从而实现与大规模数据学习相同精度的效果。我们在 ImageNet-21k 数据集上训练了不同规模的 ViT 模型，比大规模数据集 JFT-300M 上的同类模型表现更好。