Neural architecture search (NAS) has shown great promise in designing
state-of-the-art (SOTA) models that are both accurate and efficient. Recently,
two-stage NAS, e.g. BigNAS, decouples the model training and searching process
and achieves remarkable search efficiency and accuracy. Two-stage NAS requires
sampling from the search space during training, which directly impacts the
accuracy of the final searched models. While uniform sampling has been widely
used for its simplicity, it is agnostic of the model performance Pareto front,
which is the main focus in the search process, and thus, misses opportunities
to further improve the model accuracy. In this work, we propose AttentiveNAS
that focuses on improving the sampling strategy to achieve better performance
Pareto. We also propose algorithms to efficiently and effectively identify the
networks on the Pareto during training. Without extra re-training or
post-processing, we can simultaneously obtain a large number of networks across
a wide range of FLOPs. Our discovered model family, AttentiveNAS models,
achieves top-1 accuracy from 77.3% to 80.7% on ImageNet, and outperforms SOTA
models, including BigNAS and Once-for-All networks. We also achieve ImageNet
accuracy of 80.1% with only 491 MFLOPs. Our training code and pretrained models
are available at this https URL

本文提出了一种名为 AttentiveNAS 的神经结构搜索算法，通过优化网络搜索策略，实现了更好的性能 Pareto，并成功应用于 ImageNet 上，取得了性能上的巨大提升。