Deep learning is increasingly impacting various aspects of contemporary
society. Artificial neural networks have emerged as the dominant models for
solving an expanding range of tasks. The introduction of Neural Architecture
Search (NAS) techniques, which enable the automatic design of task-optimal
networks, has led to remarkable advances. However, the NAS process is typically
associated with long execution times and significant computational resource
requirements. Once-For-All (OFA) and its successor, Once-For-All-2 (OFAv2),
have been developed to mitigate these challenges. While maintaining exceptional
performance and eliminating the need for retraining, they aim to build a single
super-network model capable of directly extracting sub-networks satisfying
different constraints. Neural Architecture Transfer (NAT) was developed to
maximise the effectiveness of extracting sub-networks from a super-network. In
this paper, we present NATv2, an extension of NAT that improves multi-objective
search algorithms applied to dynamic super-network architectures. NATv2
achieves qualitative improvements in the extractable sub-networks by exploiting
the improved super-networks generated by OFAv2 and incorporating new policies
for initialisation, pre-processing and updating its networks archive. In
addition, a post-processing pipeline based on fine-tuning is introduced.
Experimental results show that NATv2 successfully improves NAT and is highly
recommended for investigating high-performance architectures with a minimal
number of parameters.

本文提出了 NATv2，它是一种改进的神经结构搜索算法，能够改善超网络结构的生成和抽取子网络的性能，并引入精调的后处理流水线，成功地提高了 NAT 的质量，因此在使用最少的参数研究高性能结构时是高度推荐的。

神经结构迁移 2：提高多目标神经结构搜索效率的范式

Neural Architecture Transfer 2: A Paradigm for Improving Efficiency in  Multi-Objective Neural Architecture Search

Neural architecture search (NAS) has emerged as a promising avenue for
automatically designing task-specific neural networks. Existing NAS approaches
require one complete search for each deployment specification of hardware or
objective. This is a computationally impractical endeavor given the potentially
large number of application scenarios. In this paper, we propose Neural
Architecture Transfer (NAT) to overcome this limitation. NAT is designed to
efficiently generate task-specific custom models that are competitive under
multiple conflicting objectives. To realize this goal we learn task-specific
supernets from which specialized subnets can be sampled without any additional
training. The key to our approach is an integrated online transfer learning and
many-objective evolutionary search procedure. A pre-trained supernet is
iteratively adapted while simultaneously searching for task-specific subnets.
We demonstrate the efficacy of NAT on 11 benchmark image classification tasks
ranging from large-scale multi-class to small-scale fine-grained datasets. In
all cases, including ImageNet, NATNets improve upon the state-of-the-art under
mobile settings ($\leq$ 600M Multiply-Adds). Surprisingly, small-scale
fine-grained datasets benefit the most from NAT. At the same time, the
architecture search and transfer is orders of magnitude more efficient than
existing NAS methods. Overall, the experimental evaluation indicates that,
across diverse image classification tasks and computational objectives, NAT is
an appreciably more effective alternative to conventional transfer learning of
fine-tuning weights of an existing network architecture learned on standard
datasets. Code is available at
this https URL

本文介绍了一种基于神经架构搜索（NAS）的新方法，称为神经架构传递（NAT），该方法可将已训练的超网络集成到在线迁移学习和多目标进化搜索流程中，以快速生成适用于多个应用场景的任务特定定制模型，并在 11 项基准图像分类任务中证明了其有效性。