One-shot Neural architecture search (One-shot NAS) has been proposed as a
time-efficient approach to obtain optimal subnet architectures and weights
under different complexity cases by training only once. However, the subnet
performance obtained by weight sharing is often inferior to the performance
achieved by retraining. In this paper, we investigate the performance gap and
attribute it to the use of uniform sampling, which is a common approach in
supernet training. Uniform sampling concentrates training resources on subnets
with intermediate computational resources, which are sampled with high
probability. However, subnets with different complexity regions require
different optimal training strategies for optimal performance. To address the
problem of uniform sampling, we propose ShiftNAS, a method that can adjust the
sampling probability based on the complexity of subnets. We achieve this by
evaluating the performance variation of subnets with different complexity and
designing an architecture generator that can accurately and efficiently provide
subnets with the desired complexity. Both the sampling probability and the
architecture generator can be trained end-to-end in a gradient-based manner.
With ShiftNAS, we can directly obtain the optimal model architecture and
parameters for a given computational complexity. We evaluate our approach on
multiple visual network models, including convolutional neural networks (CNNs)
and vision transformers (ViTs), and demonstrate that ShiftNAS is
model-agnostic. Experimental results on ImageNet show that ShiftNAS can improve
the performance of one-shot NAS without additional consumption. Source codes
are available at this https URL

本文提出 ShiftNAS，一种基于子网复杂度调整采样概率的方法，通过训练可以直接获取给定计算复杂度所需的最佳模型架构和参数，实验结果表明 ShiftNAS 可以在不增加额外消耗的情况下提高 One-shot NAS 的性能。