In recent years, zero-cost proxies are gaining ground in neural architecture
search (NAS). These methods allow finding the optimal neural network for a
given task faster and with a lesser computational load than conventional NAS
methods. Equally important is the fact that they also shed some light on the
internal workings of neural architectures. This paper presents a zero-cost
metric that highly correlates with the train set accuracy across the
NAS-Bench-101, NAS-Bench-201 and NAS-Bench-NLP benchmark datasets.
Architectures are initialised with two distinct constant shared weights, one at
a time. Then, a fixed random mini-batch of data is passed forward through each
initialisation. We observe that the dispersion of the outputs between two
initialisations positively correlates with trained accuracy. The correlation
further improves when we normalise dispersion by average output magnitude. Our
metric, epsilon, does not require gradients computation or labels. It thus
unbinds the NAS procedure from training hyperparameters, loss metrics and
human-labelled data. Our method is easy to integrate within existing NAS
algorithms and takes a fraction of a second to evaluate a single network.

该研究提出了一种与训练集准确度高度相关的零成本度量标准 epsilon，使用零成本代理来加速神经架构搜索，省去了计算负载和梯度计算、标签、损失度量等依赖于人工标注数据的环节。

轻量化且准确：通过两个常数共享权重初始化的神经架构搜索

Light and Accurate: Neural Architecture Search via Two Constant Shared Weights Initialisations

While early research in neural architecture search (NAS) required extreme
computational resources, the recent releases of tabular and surrogate
benchmarks have greatly increased the speed and reproducibility of NAS
research. However, two of the most popular benchmarks do not provide the full
training information for each architecture. As a result, on these benchmarks it
is not possible to run many types of multi-fidelity techniques, such as
learning curve extrapolation, that require evaluating architectures at
arbitrary epochs. In this work, we present a method using singular value
decomposition and noise modeling to create surrogate benchmarks, NAS-Bench-111,
NAS-Bench-311, and NAS-Bench-NLP11, that output the full training information
for each architecture, rather than just the final validation accuracy. We
demonstrate the power of using the full training information by introducing a
learning curve extrapolation framework to modify single-fidelity algorithms,
showing that it leads to improvements over popular single-fidelity algorithms
which claimed to be state-of-the-art upon release. Our code and pretrained
models are available at this https URL

本论文介绍了一种新的方法，使用奇异值分解和噪声建模来创建代理基准，该方法可以替代缺乏完整训练信息的流行基准，如 NAS-Bench-111、NAS-Bench-311 和 NAS-Bench-NLP11，从而实现对多保真技术的评估。展示了使用全面的训练信息修改单保真算法的学习曲线外推框架的强大能力，并表明这种方法比当时声称是最先进的保真算法有更好的表现。

NAS-Bench-x11 与学习曲线的能力

NAS-Bench-x11 and the Power of Learning Curves

Neural architecture search (NAS) promises to make deep learning accessible to
non-experts by automating architecture engineering of deep neural networks.
BANANAS is one state-of-the-art NAS method that is embedded within the Bayesian
optimization framework. Recent experimental findings have demonstrated the
strong performance of BANANAS on the NAS-Bench-101 benchmark being determined
by its path encoding and not its choice of surrogate model. We present
experimental results suggesting that the performance of BANANAS on the
NAS-Bench-301 benchmark is determined by its acquisition function optimizer,
which minimally mutates the incumbent.

本文介绍了最新的神经架构搜索方法 BANANAS，并通过实验结果表明这种方法在 NAS-Bench-101 基准测试中表现良好，主要是由其路径编码决定；而在 NAS-Bench-301 基准测试上表现良好的原因主要是由其采集函数优化器决定的，该优化器不会对已有的模型进行太多的改变。