One-shot Neural architecture search (One-shot NAS) has been proposed as a
time-efficient approach to obtain optimal subnet architectures and weights
under different complexity cases by training only once. However, the subnet
performance obtained by weight sharing is often inferior to the performance
achieved by retraining. In this paper, we investigate the performance gap and
attribute it to the use of uniform sampling, which is a common approach in
supernet training. Uniform sampling concentrates training resources on subnets
with intermediate computational resources, which are sampled with high
probability. However, subnets with different complexity regions require
different optimal training strategies for optimal performance. To address the
problem of uniform sampling, we propose ShiftNAS, a method that can adjust the
sampling probability based on the complexity of subnets. We achieve this by
evaluating the performance variation of subnets with different complexity and
designing an architecture generator that can accurately and efficiently provide
subnets with the desired complexity. Both the sampling probability and the
architecture generator can be trained end-to-end in a gradient-based manner.
With ShiftNAS, we can directly obtain the optimal model architecture and
parameters for a given computational complexity. We evaluate our approach on
multiple visual network models, including convolutional neural networks (CNNs)
and vision transformers (ViTs), and demonstrate that ShiftNAS is
model-agnostic. Experimental results on ImageNet show that ShiftNAS can improve
the performance of one-shot NAS without additional consumption. Source codes
are available at this https URL

本文提出 ShiftNAS，一种基于子网复杂度调整采样概率的方法，通过训练可以直接获取给定计算复杂度所需的最佳模型架构和参数，实验结果表明 ShiftNAS 可以在不增加额外消耗的情况下提高 One-shot NAS 的性能。

通过概率变化改进一次性 NAS 的 ShiftNAS

ShiftNAS: Improving One-shot NAS via Probability Shift

In one-shot NAS, sub-networks need to be searched from the supernet to meet
different hardware constraints. However, the search cost is high and $N$ times
of searches are needed for $N$ different constraints. In this work, we propose
a novel search strategy called architecture generator to search sub-networks by
generating them, so that the search process can be much more efficient and
flexible. With the trained architecture generator, given target hardware
constraints as the input, $N$ good architectures can be generated for $N$
constraints by just one forward pass without re-searching and supernet
retraining. Moreover, we propose a novel single-path supernet, called unified
supernet, to further improve search efficiency and reduce GPU memory
consumption of the architecture generator. With the architecture generator and
the unified supernet, we propose a flexible and efficient one-shot NAS
framework, called Searching by Generating NAS (SGNAS). With the pre-trained
supernt, the search time of SGNAS for $N$ different hardware constraints is
only 5 GPU hours, which is $4N$ times faster than previous SOTA single-path
methods. After training from scratch, the top1-accuracy of SGNAS on ImageNet is
77.1%, which is comparable with the SOTAs. The code is available at:
this https URL

提出一种名为 SGNAS 的一次搜索神经结构的框架，使用生成器生成子网络以更加高效和灵活地进行搜索，同时还提出了一种名为统一超网络的新单路径超网络以进一步提高搜索效率和减少 GPU 内存消耗，在 ImageNet 数据集上仅用 5 个 GPU 小时完成 $N$ 个不同硬件限制下的搜索，准确率达 77.1%。