Vision Transformer (ViT) demonstrates that Transformer for natural language
processing can be applied to computer vision tasks and result in comparable
performance to convolutional neural networks (CNN), which have been studied and
adopted in computer vision for years. This naturally raises the question of how
the performance of ViT can be advanced with design techniques of CNN. To this
end, we propose to incorporate two techniques and present ViT-ResNAS, an
efficient multi-stage ViT architecture designed with neural architecture search
(NAS). First, we propose residual spatial reduction to decrease sequence
lengths for deeper layers and utilize a multi-stage architecture. When reducing
lengths, we add skip connections to improve performance and stabilize training
deeper networks. Second, we propose weight-sharing NAS with multi-architectural
sampling. We enlarge a network and utilize its sub-networks to define a search
space. A super-network covering all sub-networks is then trained for fast
evaluation of their performance. To efficiently train the super-network, we
propose to sample and train multiple sub-networks with one forward-backward
pass. After that, evolutionary search is performed to discover high-performance
network architectures. Experiments on ImageNet demonstrate that ViT-ResNAS
achieves better accuracy-MACs and accuracy-throughput trade-offs than the
original DeiT and other strong baselines of ViT. Code is available at
this https URL

利用神经架构搜索（NAS）设计了一个有效的多阶段的 Vision Transformer 架构 ViT-ResNAS，其中融合了两个技术：残差空间缩减和权重共享 NAS，实验证明 ViT-ResNAS 在 ImageNet 数据集上能够取得比原始 DeiT 和其他强基线更好的精度 - MAC 和精度 - 吞吐量权衡。