Efficient performance estimation of architectures drawn from large search
spaces is essential to Neural Architecture Search. One-Shot methods tackle this
challenge by training one supernet to approximate the performance of every
architecture in the search space via weight-sharing, thereby drastically
reducing the search cost. However, due to coupled optimization between child
architectures caused by weight-sharing, One-Shot supernet's performance
estimation could be inaccurate, leading to degraded search outcomes. To address
this issue, Few-Shot NAS reduces the level of weight-sharing by splitting the
One-Shot supernet into multiple separated sub-supernets via edge-wise
(layer-wise) exhaustive partitioning. Since each partition of the supernet is
not equally important, it necessitates the design of a more effective splitting
criterion. In this work, we propose a gradient matching score (GM) that
leverages gradient information at the shared weight for making informed
splitting decisions. Intuitively, gradients from different child models can be
used to identify whether they agree on how to update the shared modules, and
subsequently to decide if they should share the same weight. Compared with
exhaustive partitioning, the proposed criterion significantly reduces the
branching factor per edge. This allows us to split more edges (layers) for a
given budget, resulting in substantially improved performance as NAS search
spaces usually include dozens of edges (layers). Extensive empirical
evaluations of the proposed method on a wide range of search spaces
(NASBench-201, DARTS, MobileNet Space), datasets (cifar10, cifar100, ImageNet)
and search algorithms (DARTS, SNAS, RSPS, ProxylessNAS, OFA) demonstrate that
it significantly outperforms its Few-Shot counterparts while surpassing
previous comparable methods in terms of the accuracy of derived architectures.

该研究提出了一种基于梯度匹配得分的 Few-Shot NAS 方法，通过对多个超网进行边级分区来减小权重共享带来的误差，极大地减少了搜索成本，并在各种数据集、搜索算法下展开的广泛实证评估表明，该方法在搜索效果方面有明显的提高。

基于梯度匹配的 Few-Shot NAS 泛化

Generalizing Few-Shot NAS with Gradient Matching

Efficient evaluation of a network architecture drawn from a large search
space remains a key challenge in Neural Architecture Search (NAS). Vanilla NAS
evaluates each architecture by training from scratch, which gives the true
performance but is extremely time-consuming. Recently, one-shot NAS
substantially reduces the computation cost by training only one supernetwork,
a.k.a. supernet, to approximate the performance of every architecture in the
search space via weight-sharing. However, the performance estimation can be
very inaccurate due to the co-adaption among operations. In this paper, we
propose few-shot NAS that uses multiple supernetworks, called sub-supernet,
each covering different regions of the search space to alleviate the undesired
co-adaption. Compared to one-shot NAS, few-shot NAS improves the accuracy of
architecture evaluation with a small increase of evaluation cost. With only up
to 7 sub-supernets, few-shot NAS establishes new SoTAs: on ImageNet, it finds
models that reach 80.5% top-1 accuracy at 600 MB FLOPS and 77.5% top-1 accuracy
at 238 MFLOPS; on CIFAR10, it reaches 98.72% top-1 accuracy without using extra
data or transfer learning. In Auto-GAN, few-shot NAS outperforms the previously
published results by up to 20%. Extensive experiments show that few-shot NAS
significantly improves various one-shot methods, including 4 gradient-based and
6 search-based methods on 3 different tasks in NasBench-201 and
NasBench1-shot-1.

该研究提出了 few-shot NAS，利用多个超级网络（称为子超级网络）覆盖搜索空间的不同区域，以缓解操作之间的不良共适应，并证明其在多项任务中能够显著提高各种一次性方法的准确性，包括在 NasBench-201 和 NasBench1-shot-1 上的 3 个不同任务中的 4 个基于梯度的和 6 个基于搜索的方法，并在 ImageNet、CIFAR10 和 Auto-GAN 上取得最新水平。