One-shot neural architecture search (NAS) has played a crucial role in making
NAS methods computationally feasible in practice. Nevertheless, there is still
a lack of understanding on how these weight-sharing algorithms exactly work due
to the many factors controlling the dynamics of the process. In order to allow
a scientific study of these components, we introduce a general framework for
one-shot NAS that can be instantiated to many recently-introduced variants and
introduce a general benchmarking framework that draws on the recent large-scale
tabular benchmark NAS-Bench-101 for cheap anytime evaluations of one-shot NAS
methods. To showcase the framework, we compare several state-of-the-art
one-shot NAS methods, examine how sensitive they are to their hyperparameters
and how they can be improved by tuning their hyperparameters, and compare their
performance to that of blackbox optimizers for NAS-Bench-101.

本文提出了一个通用框架，用于一次性神经架构搜索，并引入了一个基于最近推出的大规模表格基准 NAS-Bench-101 的基准测试框架，以评估一次性 NAS 方法的廉价性能，并比较了几种最先进的一次性 NAS 方法，检查它们对超参数的敏感性，以及如何通过调整它们的超参数来改进它们的性能，并将它们的性能与 NAS-Bench-101 的黑盒优化器进行比较。

NAS-Bench-1Shot1: 一次性神经架构搜索的基准测试和解剖

NAS-Bench-1Shot1: Benchmarking and Dissecting One-shot Neural  Architecture Search

Recent advances in neural architecture search (NAS) demand tremendous
computational resources, which makes it difficult to reproduce experiments and
imposes a barrier-to-entry to researchers without access to large-scale
computation. We aim to ameliorate these problems by introducing NAS-Bench-101,
the first public architecture dataset for NAS research. To build NAS-Bench-101,
we carefully constructed a compact, yet expressive, search space, exploiting
graph isomorphisms to identify 423k unique convolutional architectures. We
trained and evaluated all of these architectures multiple times on CIFAR-10 and
compiled the results into a large dataset of over 5 million trained models.
This allows researchers to evaluate the quality of a diverse range of models in
milliseconds by querying the pre-computed dataset. We demonstrate its utility
by analyzing the dataset as a whole and by benchmarking a range of architecture
optimization algorithms.

本文介绍了 NAS-Bench-101，这是第一个公开的神经架构搜索数据集，并描述了它是如何通过在 CIFAR-10 上训练和评估超过 423k 个卷积架构来构建的，旨在帮助研究者在毫秒内查询预计算的数据集来评估各种模型的质量。此外，作者还展示了该数据集作为整体的可用性以及对一系列架构优化算法进行基准测试的实用性。