Transformers have been recently adapted for large scale image classification, achieving high scores shaking up the long supremacy of convolutional neural networks. However the optimization of image transformers has been little studied so far. In this work, we build and optimize deeper transformer networks for image classification. In particular, we investigate the interplay of architecture and optimization of such dedicated transformers. We make two transformers architecture changes that significantly improve the accuracy of deep transformers. This leads us to produce models whose performance does not saturate early with more depth, for instance we obtain 86.3% top-1 accuracy on Imagenet when training with no external data. Our best model establishes the new state of the art on Imagenet with Reassessed labels and Imagenet-V2 / match frequency, in the setting with no additional training data.

本文研究了基于Transformer的图片分类模型的优化，通过两个Transformer模型的改进，使得模型深度增加能够带来更好的性能表现，并在Imagenet数据集上取得了86.5%的top-1准确率，创造了当前最高成绩。同时，我们还通过重新评估标签，打破了Imagenet-V2数据集的最高准确率记录，并开放了源代码和训练好的模型。

图像Transformer的深入研究