Efficient performance estimation of architectures drawn from large search
spaces is essential to Neural Architecture Search. One-Shot methods tackle this
challenge by training one supernet to approximate the performance of every
architecture in the search space via weight-sharing, thereby drastically
reducing the search cost. However, due to coupled optimization between child
architectures caused by weight-sharing, One-Shot supernet's performance
estimation could be inaccurate, leading to degraded search outcomes. To address
this issue, Few-Shot NAS reduces the level of weight-sharing by splitting the
One-Shot supernet into multiple separated sub-supernets via edge-wise
(layer-wise) exhaustive partitioning. Since each partition of the supernet is
not equally important, it necessitates the design of a more effective splitting
criterion. In this work, we propose a gradient matching score (GM) that
leverages gradient information at the shared weight for making informed
splitting decisions. Intuitively, gradients from different child models can be
used to identify whether they agree on how to update the shared modules, and
subsequently to decide if they should share the same weight. Compared with
exhaustive partitioning, the proposed criterion significantly reduces the
branching factor per edge. This allows us to split more edges (layers) for a
given budget, resulting in substantially improved performance as NAS search
spaces usually include dozens of edges (layers). Extensive empirical
evaluations of the proposed method on a wide range of search spaces
(NASBench-201, DARTS, MobileNet Space), datasets (cifar10, cifar100, ImageNet)
and search algorithms (DARTS, SNAS, RSPS, ProxylessNAS, OFA) demonstrate that
it significantly outperforms its Few-Shot counterparts while surpassing
previous comparable methods in terms of the accuracy of derived architectures.

该研究提出了一种基于梯度匹配得分的 Few-Shot NAS 方法，通过对多个超网进行边级分区来减小权重共享带来的误差，极大地减少了搜索成本，并在各种数据集、搜索算法下展开的广泛实证评估表明，该方法在搜索效果方面有明显的提高。