DARTS search space (DSS) has become a canonical benchmark for NAS whereas
some emerging works pointed out the issue of narrow accuracy range and claimed
it would hurt the method ranking. We observe some recent studies already suffer
from this issue that overshadows the meaning of scores. In this work, we first
propose and orchestrate a suite of improvements to frame a larger and harder
DSS, termed LHD, while retaining high efficiency in search. We step forward to
renovate a LHD-based new benchmark, taking care of both discernibility and
accessibility. Specifically, we re-implement twelve baselines and evaluate them
across twelve conditions by combining two underexpolored influential factors:
transductive robustness and discretization policy, to reasonably construct a
benchmark upon multi-condition evaluation. Considering that the tabular
benchmarks are always insufficient to adequately evaluate the methods of neural
architecture search (NAS), our work can serve as a crucial basis for the future
progress of NAS. this https URL

本文提出了一种改进的、更大更难的基于 DARTS search space (LHD) 的基准测试，重新实现了 12 个基线并在 12 个条件下进行了评估，并探究了多种情况下的影响因素，可以作为神经结构搜索 (NAS) 未来进展的关键依据。

重新思考 DARTS 搜索空间并建立新的基准测试

Rethink DARTS Search Space and Renovate a New Benchmark

Neural Architecture Search (NAS) is an exciting new field which promises to
be as much as a game-changer as Convolutional Neural Networks were in 2012.
Despite many great works leading to substantial improvements on a variety of
tasks, comparison between different methods is still very much an open issue.
While most algorithms are tested on the same datasets, there is no shared
experimental protocol followed by all. As such, and due to the under-use of
ablation studies, there is a lack of clarity regarding why certain methods are
more effective than others. Our first contribution is a benchmark of $8$ NAS
methods on $5$ datasets. To overcome the hurdle of comparing methods with
different search spaces, we propose using a method's relative improvement over
the randomly sampled average architecture, which effectively removes advantages
arising from expertly engineered search spaces or training protocols.
Surprisingly, we find that many NAS techniques struggle to significantly beat
the average architecture baseline. We perform further experiments with the
commonly used DARTS search space in order to understand the contribution of
each component in the NAS pipeline. These experiments highlight that: (i) the
use of tricks in the evaluation protocol has a predominant impact on the
reported performance of architectures; (ii) the cell-based search space has a
very narrow accuracy range, such that the seed has a considerable impact on
architecture rankings; (iii) the hand-designed macro-structure (cells) is more
important than the searched micro-structure (operations); and (iv) the
depth-gap is a real phenomenon, evidenced by the change in rankings between $8$
and $20$ cell architectures. To conclude, we suggest best practices, that we
hope will prove useful for the community and help mitigate current NAS
pitfalls. The code used is available at
this https URL

在本文中，我们提出了用于比较 NAS 方法的基准测试，并对常用的 DARTS search space 进行了进一步的实验。我们发现，手工设计的宏观结构比搜索到的微观结构更重要，并且深度间隔是一个真实的现象，最后我们提出了最佳实践以帮助缓解当前的 NAS 缺陷。

NAS 评估非常困难

NAS evaluation is frustratingly hard

Neural Architecture Search remains a very challenging meta-learning problem.
Several recent techniques based on parameter-sharing idea have focused on
reducing the NAS running time by leveraging proxy models, leading to
architectures with competitive performance compared to those with hand-crafted
designs. In this paper, we propose an iterative technique for NAS, inspired by
algorithms for learning low-degree sparse Boolean functions. We validate our
approach on the DARTs search space (Liu et al., 2018b) and NAS-Bench-201 (Yang
et al., 2020). In addition, we provide theoretical analysis via upper bounds on
the number of validation error measurements needed for reliable learning, and
include ablation studies to further in-depth understanding of our technique.

本文提出一种迭代学习方法，旨在解决神经架构搜索中的元学习问题。该方法使用参数共享思想，利用代理模型来减少 NAS 运行时间，并在 DARTs 搜索空间和 NAS-Bench-201 上验证了其可行性。此外，作者还提供了理论分析及脱敏研究以进一步深入了解其技术。