Supernet is a core component in many recent Neural Architecture Search (NAS)
methods. It not only helps embody the search space but also provides a
(relative) estimation of the final performance of candidate architectures.
Thus, it is critical that the top architectures ranked by a supernet should be
consistent with those ranked by true performance, which is known as the
order-preserving ability. In this work, we analyze the order-preserving ability
on the whole search space (global) and a sub-space of top architectures
(local), and empirically show that the local order-preserving for current
two-stage NAS methods still need to be improved. To rectify this, we propose a
novel concept of Supernet Shifting, a refined search strategy combining
architecture searching with supernet fine-tuning. Specifically, apart from
evaluating, the training loss is also accumulated in searching and the supernet
is updated every iteration. Since superior architectures are sampled more
frequently in evolutionary searching, the supernet is encouraged to focus on
top architectures, thus improving local order-preserving. Besides, a
pre-trained supernet is often un-reusable for one-shot methods. We show that
Supernet Shifting can fulfill transferring supernet to a new dataset.
Specifically, the last classifier layer will be unset and trained through
evolutionary searching. Comprehensive experiments show that our method has
better order-preserving ability and can find a dominating architecture.
Moreover, the pre-trained supernet can be easily transferred into a new dataset
with no loss of performance.

Supernet 是许多最新的神经架构搜索（NAS）方法中的核心组件，本文提出了一种改进的搜索策略 ——Supernet Shifting，以改善局部排序能力和实现模型的迁移学习。