Neural Architecture Search (NAS) is widely used to automatically obtain the
neural network with the best performance among a large number of candidate
architectures. To reduce the search time, zero-shot NAS aims at designing
training-free proxies that can predict the test performance of a given
architecture. However, as shown recently, none of the zero-shot proxies
proposed to date can actually work consistently better than a naive proxy,
namely, the number of network parameters (#Params). To improve this state of
affairs, as the main theoretical contribution, we first reveal how some
specific gradient properties across different samples impact the convergence
rate and generalization capacity of neural networks. Based on this theoretical
analysis, we propose a new zero-shot proxy, ZiCo, the first proxy that works
consistently better than #Params. We demonstrate that ZiCo works better than
State-Of-The-Art (SOTA) proxies on several popular NAS-Benchmarks (NASBench101,
NATSBench-SSS/TSS, TransNASBench-101) for multiple applications (e.g., image
classification/reconstruction and pixel-level prediction). Finally, we
demonstrate that the optimal architectures found via ZiCo are as competitive as
the ones found by one-shot and multi-shot NAS methods, but with much less
search time. For example, ZiCo-based NAS can find optimal architectures with
78.1%, 79.4%, and 80.4% test accuracy under inference budgets of 450M, 600M,
and 1000M FLOPs, respectively, on ImageNet within 0.4 GPU days. Our code is
available at this https URL

本研究提出了一种新的零 - shot 代理模型（ZiCo），它能够通过预测测试性能来设计最佳神经网络架构，不仅比现有模型表现更好，而且不需要训练，可以在进行神经架构搜索时节省大量时间。此外，通过在 NAS-Benchmarks 基准下的对比实验，我们证明了该方法对于多种应用的数据集具有良好的适应性，并且所得到的最佳神经网络架构不仅与传统 NAS 方法相当，而且所需搜索时间更短。