Neural architecture search (NAS) has achieved breakthrough success in a great
number of applications in the past few years. It could be time to take a step
back and analyze the good and bad aspects in the field of NAS. A variety of
algorithms search architectures under different search space. These searched
architectures are trained using different setups, e.g., hyper-parameters, data
augmentation, regularization. This raises a comparability problem when
comparing the performance of various NAS algorithms. NAS-Bench-101 has shown
success to alleviate this problem. In this work, we propose an extension to
NAS-Bench-101: NAS-Bench-201 with a different search space, results on multiple
datasets, and more diagnostic information. NAS-Bench-201 has a fixed search
space and provides a unified benchmark for almost any up-to-date NAS
algorithms. The design of our search space is inspired from the one used in the
most popular cell-based searching algorithms, where a cell is represented as a
DAG. Each edge here is associated with an operation selected from a predefined
operation set. For it to be applicable for all NAS algorithms, the search space
defined in NAS-Bench-201 includes all possible architectures generated by 4
nodes and 5 associated operation options, which results in 15,625 candidates in
total. The training log and the performance for each architecture candidate are
provided for three datasets. This allows researchers to avoid unnecessary
repetitive training for selected candidate and focus solely on the search
algorithm itself. The training time saved for every candidate also largely
improves the efficiency of many methods. We provide additional diagnostic
information such as fine-grained loss and accuracy, which can give inspirations
to new designs of NAS algorithms. In further support, we have analyzed it from
many aspects and benchmarked 10 recent NAS algorithms.

提出 NAS-Bench-201，通过设计新的搜索空间和提供多个数据集的结果以及诊断信息，为几乎所有最新的 NAS 算法提供一个统一的基准，从而避免了重复训练，并提高了搜索算法的效率。

NAS-Bench-201: 扩展可复现神经架构搜索的范围

NAS-Bench-201: Extending the Scope of Reproducible Neural Architecture  Search

Neural Architecture Search remains a very challenging meta-learning problem.
Several recent techniques based on parameter-sharing idea have focused on
reducing the NAS running time by leveraging proxy models, leading to
architectures with competitive performance compared to those with hand-crafted
designs. In this paper, we propose an iterative technique for NAS, inspired by
algorithms for learning low-degree sparse Boolean functions. We validate our
approach on the DARTs search space (Liu et al., 2018b) and NAS-Bench-201 (Yang
et al., 2020). In addition, we provide theoretical analysis via upper bounds on
the number of validation error measurements needed for reliable learning, and
include ablation studies to further in-depth understanding of our technique.

本文提出一种迭代学习方法，旨在解决神经架构搜索中的元学习问题。该方法使用参数共享思想，利用代理模型来减少 NAS 运行时间，并在 DARTs 搜索空间和 NAS-Bench-201 上验证了其可行性。此外，作者还提供了理论分析及脱敏研究以进一步深入了解其技术。