This work targets automated designing and scaling of Vision Transformers
(ViTs). The motivation comes from two pain spots: 1) the lack of efficient and
principled methods for designing and scaling ViTs; 2) the tremendous
computational cost of training ViT that is much heavier than its convolution
counterpart. To tackle these issues, we propose As-ViT, an auto-scaling
framework for ViTs without training, which automatically discovers and scales
up ViTs in an efficient and principled manner. Specifically, we first design a
"seed" ViT topology by leveraging a training-free search process. This
extremely fast search is fulfilled by a comprehensive study of ViT's network
complexity, yielding a strong Kendall-tau correlation with ground-truth
accuracies. Second, starting from the "seed" topology, we automate the scaling
rule for ViTs by growing widths/depths to different ViT layers. This results in
a series of architectures with different numbers of parameters in a single run.
Finally, based on the observation that ViTs can tolerate coarse tokenization in
early training stages, we propose a progressive tokenization strategy to train
ViTs faster and cheaper. As a unified framework, As-ViT achieves strong
performance on classification (83.5% top1 on ImageNet-1k) and detection (52.7%
mAP on COCO) without any manual crafting nor scaling of ViT architectures: the
end-to-end model design and scaling process cost only 12 hours on one V100 GPU.
Our code is available at this https URL

提出了一种名为 As-ViT 的自动缩放框架，用于设计和扩展 Vision Transformers (ViT)，并在分类和检测任务上获得了强大的性能，其模型设计和缩放过程仅需 12 小时的训练。