Training-free Vision Transformer (ViT) architecture search is presented to
search for a better ViT with zero-cost proxies. While ViTs achieve significant
distillation gains from CNN teacher models on small datasets, the current
zero-cost proxies in ViTs do not generalize well to the distillation training
paradigm according to our experimental observations. In this paper, for the
first time, we investigate how to search in a training-free manner with the
help of teacher models and devise an effective Training-free ViT (TVT) search
framework. Firstly, we observe that the similarity of attention maps between
ViT and ConvNet teachers affects distill accuracy notably. Thus, we present a
teacher-aware metric conditioned on the feature attention relations between
teacher and student. Additionally, TVT employs the L2-Norm of the student's
weights as the student-capability metric to improve ranking consistency.
Finally, TVT searches for the best ViT for distilling with ConvNet teachers via
our teacher-aware metric and student-capability metric, resulting in impressive
gains in efficiency and effectiveness. Extensive experiments on various tiny
datasets and search spaces show that our TVT outperforms state-of-the-art
training-free search methods. The code will be released.

本文提出了一种无需训练的视觉 Transformer（ViT）架构搜索方法，通过使用教师模型，利用具有零成本代理的方式搜索更好的 ViT，并在各种小型数据集和搜索空间上展开了大量实验，证明了该方法优于现有的无训练搜索方法。