Existing neural architecture search (NAS) methods typically rely on
pre-specified super deep neural networks (super-networks) with handcrafted
search spaces beforehand. Such requirements make it challenging to extend them
onto general scenarios without significant human expertise and manual
intervention. To overcome the limitations, we propose the third generation of
Only-Train-Once (OTOv3). OTOv3 is perhaps the first automated system that
trains general super-networks and produces high-performing sub-networks in the
one shot manner without pretraining and fine-tuning. Technologically, OTOv3
delivers three noticeable contributions to minimize human efforts: (i)
automatic search space construction for general super-networks; (ii) a
Hierarchical Half-Space Projected Gradient (H2SPG) that leverages the
dependency graph to ensure the network validity during optimization and
reliably produces a solution with both high performance and hierarchical group
sparsity; and (iii) automatic sub-network construction based on the
super-network and the H2SPG solution. Numerically, we demonstrate the
effectiveness of OTOv3 on a variety of super-networks, including RegNet,
StackedUnets, SuperResNet, and DARTS, over benchmark datasets such as CIFAR10,
Fashion-MNIST, ImageNet, STL-10, and SVNH. The sub-networks computed by OTOv3
achieve competitive even superior performance compared to the super-networks
and other state-of-the-arts. The library will be released at
this https URL

提出第三代 “只训练一次（OTOv3）” 自动化系统，在不需要人工手工干预的情况下训练通用超级网络，并以单次方式生成高性能子网络，该系统在广泛的超级网络上实现了很好的优化性能，并且相对于超级网络及其他技术水平优越的性能。

通往通用超级网络的自动神经架构搜索

Towards Automatic Neural Architecture Search within General  Super-Networks

Recent Neural Architecture Search (NAS) solutions have produced impressive
results training super-networks and then deriving subnetworks, a.k.a. child
models that outperform expert-crafted models from a pre-defined search space.
Efficient and robust subnetworks can be selected for resource-constrained edge
devices, allowing them to perform well in the wild. However, constructing
super-networks for arbitrary architectures is still a challenge that often
prevents the adoption of these approaches. To address this challenge, we
present BootstrapNAS, a software framework for automatic generation of
super-networks for NAS. BootstrapNAS takes a pre-trained model from a popular
architecture, e.g., ResNet- 50, or from a valid custom design, and
automatically creates a super-network out of it, then uses state-of-the-art NAS
techniques to train the super-network, resulting in subnetworks that
significantly outperform the given pre-trained model. We demonstrate the
solution by generating super-networks from arbitrary model repositories and
make available the resulting super-networks for reproducibility of the results.

本文描述了 BootstrapNAS，这是一个用于自动生成 NAS 超网络的软件框架，它基于预先训练的模型创建超网络，然后使用先进的 NAS 技术来训练超网络，生成有效的子网络，为资源受限的边缘设备提供高效解决方案。