Conventional wisdom suggests that pre-training Vision Transformers (ViT) improves downstream performance by learning useful representations. Is this actually true? We investigate this question and find that the features and representations learned during pre-training are not essential. Surprisingly, using only the attention patterns from pre-training (i.e., guiding how information flows between tokens) is sufficient for models to learn high quality features from scratch and achieve comparable downstream performance. We show this by introducing a simple method called attention transfer, where only the attention patterns from a pre-trained teacher ViT are transferred to a student, either by copying or distilling the attention maps. Since attention transfer lets the student learn its own features, ensembling it with a fine-tuned teacher also further improves accuracy on ImageNet. We systematically study various aspects of our findings on the sufficiency of attention maps, including distribution shift settings where they underperform fine-tuning. We hope our exploration provides a better understanding of what pre-training accomplishes and leads to a useful alternative to the standard practice of fine-tuning

本研究关注预训练视觉变换器（ViT）对下游性能影响的真实性，发现预训练过程中学习的特征并非必需。研究提出了一种简单的方法——注意力转移，表明仅通过转移预训练教师模型的注意力模式，学生模型也能从零开始学习出高质量特征，并实现相当的下游性能。这一发现为理解预训练的作用提供了新的视角，并为传统的微调方法提供了替代方案。

注意力转移在视觉变换器中的惊人有效性