Neural architecture search methods seek optimal candidates with efficient
weight-sharing supernet training. However, recent studies indicate poor ranking
consistency about the performance between stand-alone architectures and
shared-weight networks. In this paper, we present Prior-Guided One-shot NAS
(PGONAS) to strengthen the ranking correlation of supernets. Specifically, we
first explore the effect of activation functions and propose a balanced
sampling strategy based on the Sandwich Rule to alleviate weight coupling in
the supernet. Then, FLOPs and Zen-Score are adopted to guide the training of
supernet with ranking correlation loss. Our PGONAS ranks 3rd place in the
supernet Track Track of CVPR2022 Second lightweight NAS challenge. Code is
available in
this https URL

本文介绍一种名为 Prior-Guided One-shot NAS 的神经架构搜索方法，它采用平衡采样策略、FLOPs 和 Zen-Score 等机制来加强超级网络的排名相关性，取得了 CVPR2022 第二轻量化 NAS 挑战赛超级网络轨道的第三名。

基于先验指导的一次性神经架构搜索

Prior-Guided One-shot Neural Architecture Search

Recently proposed neural architecture search (NAS) methods co-train billions
of architectures in a supernet and estimate their potential accuracy using the
network weights detached from the supernet. However, the ranking correlation
between the architectures' predicted accuracy and their actual capability is
incorrect, which causes the existing NAS methods' dilemma. We attribute this
ranking correlation problem to the supernet training consistency shift,
including feature shift and parameter shift. Feature shift is identified as
dynamic input distributions of a hidden layer due to random path sampling. The
input distribution dynamic affects the loss descent and finally affects
architecture ranking. Parameter shift is identified as contradictory parameter
updates for a shared layer lay in different paths in different training steps.
The rapidly-changing parameter could not preserve architecture ranking. We
address these two shifts simultaneously using a nontrivial supernet-Pi model,
called Pi-NAS. Specifically, we employ a supernet-Pi model that contains
cross-path learning to reduce the feature consistency shift between different
paths. Meanwhile, we adopt a novel nontrivial mean teacher containing negative
samples to overcome parameter shift and model collision. Furthermore, our
Pi-NAS runs in an unsupervised manner, which can search for more transferable
architectures. Extensive experiments on ImageNet and a wide range of downstream
tasks (e.g., COCO 2017, ADE20K, and Cityscapes) demonstrate the effectiveness
and universality of our Pi-NAS compared to supervised NAS. See Codes:
this https URL

Pi-NAS 是一种非平凡的超网 - Pi 模型，利用跨路径学习减少不同路径之间特征的一致性移位并采用新颖的平均教师包含负样本克服参数移位和模型冲突，在不监督的情况下搜索可转移架构，并在 ImageNet 和一系列下游任务上展示了其相对于受监督的 NAS 的有效性和普适性。