Recently, neural networks purely based on attention were shown to address
image understanding tasks such as image classification. However, these visual
transformers are pre-trained with hundreds of millions of images using an
expensive infrastructure, thereby limiting their adoption.
In this work, we produce a competitive convolution-free transformer by
training on Imagenet only. We train them on a single computer in less than 3
days. Our reference vision transformer (86M parameters) achieves top-1 accuracy
of 83.1% (single-crop evaluation) on ImageNet with no external data.
More importantly, we introduce a teacher-student strategy specific to
transformers. It relies on a distillation token ensuring that the student
learns from the teacher through attention. We show the interest of this
token-based distillation, especially when using a convnet as a teacher. This
leads us to report results competitive with convnets for both Imagenet (where
we obtain up to 85.2% accuracy) and when transferring to other tasks. We share
our code and models.

通过使用注意力机制的神经网络，我们以卷积神经网络为教师，在 ImageNet 上训练单个计算机下的优秀可比拟的转换器，并引入了基于蒸馏令牌的教师 - 学生策略，以获得竞争性结果。