One-shot Neural architecture search (One-shot NAS) has been proposed as a
time-efficient approach to obtain optimal subnet architectures and weights
under different complexity cases by training only once. However, the subnet
performance obtained by weight sharing is often inferior to the performance
achieved by retraining. In this paper, we investigate the performance gap and
attribute it to the use of uniform sampling, which is a common approach in
supernet training. Uniform sampling concentrates training resources on subnets
with intermediate computational resources, which are sampled with high
probability. However, subnets with different complexity regions require
different optimal training strategies for optimal performance. To address the
problem of uniform sampling, we propose ShiftNAS, a method that can adjust the
sampling probability based on the complexity of subnets. We achieve this by
evaluating the performance variation of subnets with different complexity and
designing an architecture generator that can accurately and efficiently provide
subnets with the desired complexity. Both the sampling probability and the
architecture generator can be trained end-to-end in a gradient-based manner.
With ShiftNAS, we can directly obtain the optimal model architecture and
parameters for a given computational complexity. We evaluate our approach on
multiple visual network models, including convolutional neural networks (CNNs)
and vision transformers (ViTs), and demonstrate that ShiftNAS is
model-agnostic. Experimental results on ImageNet show that ShiftNAS can improve
the performance of one-shot NAS without additional consumption. Source codes
are available at this https URL

本文提出 ShiftNAS，一种基于子网复杂度调整采样概率的方法，通过训练可以直接获取给定计算复杂度所需的最佳模型架构和参数，实验结果表明 ShiftNAS 可以在不增加额外消耗的情况下提高 One-shot NAS 的性能。

通过概率变化改进一次性 NAS 的 ShiftNAS

ShiftNAS: Improving One-shot NAS via Probability Shift

We present BN-NAS, neural architecture search with Batch Normalization
(BN-NAS), to accelerate neural architecture search (NAS). BN-NAS can
significantly reduce the time required by model training and evaluation in NAS.
Specifically, for fast evaluation, we propose a BN-based indicator for
predicting subnet performance at a very early training stage. The BN-based
indicator further facilitates us to improve the training efficiency by only
training the BN parameters during the supernet training. This is based on our
observation that training the whole supernet is not necessary while training
only BN parameters accelerates network convergence for network architecture
search. Extensive experiments show that our method can significantly shorten
the time of training supernet by more than 10 times and shorten the time of
evaluating subnets by more than 600,000 times without losing accuracy.

本研究提出了使用批量归一化的神经架构搜索 BN-NAS 算法，通过训练 BN 参数加速神经架构搜索，实验证明该算法可以显著缩短训练时间和运行时间。