Neural Architecture Search (NAS), aiming at automatically designing neural
architectures by machines, has been considered a key step toward automatic
machine learning. One notable NAS branch is the weight-sharing NAS, which
significantly improves search efficiency and allows NAS algorithms to run on
ordinary computers. Despite receiving high expectations, this category of
methods suffers from low search effectiveness. By employing a generalization
boundedness tool, we demonstrate that the devil behind this drawback is the
untrustworthy architecture rating with the oversized search space of the
possible architectures. Addressing this problem, we modularize a large search
space into blocks with small search spaces and develop a family of models with
the distilling neural architecture (DNA) techniques. These proposed models,
namely a DNA family, are capable of resolving multiple dilemmas of the
weight-sharing NAS, such as scalability, efficiency, and multi-modal
compatibility. Our proposed DNA models can rate all architecture candidates, as
opposed to previous works that can only access a sub- search space using
heuristic algorithms. Moreover, under a certain computational complexity
constraint, our method can seek architectures with different depths and widths.
Extensive experimental evaluations show that our models achieve
state-of-the-art top-1 accuracy of 78.9% and 83.6% on ImageNet for a mobile
convolutional network and a small vision transformer, respectively.
Additionally, we provide in-depth empirical analysis and insights into neural
architecture ratings. Codes available: https://github.com/changlin31/DNA.

通过将大的搜索空间模块化为小的搜索空间，并利用 DNA 技术开发了一系列模型，我们解决了 weight-sharing NAS 中的多个困境，如可扩展性、效率和多模态兼容性，并在一定计算复杂度约束下寻求不同深度和宽度的架构。广泛的实验评估表明，我们的模型分别在移动卷积网络和小型视觉变换器上实现了 78.9％和 83.6％的 ImageNet 最高准确率。