Neural Architecture Search (NAS) has emerged as one of the effective methods
to design the optimal neural network architecture automatically. Although
neural architectures have achieved human-level performances in several tasks,
few of them are obtained from the NAS method. The main reason is the huge
search space of neural architectures, making NAS algorithms inefficient. This
work presents a novel architecture search algorithm, called GPT-NAS, that
optimizes neural architectures by Generative Pre-Trained (GPT) model. In
GPT-NAS, we assume that a generative model pre-trained on a large-scale corpus
could learn the fundamental law of building neural architectures. Therefore,
GPT-NAS leverages the generative pre-trained (GPT) model to propose reasonable
architecture components given the basic one. Such an approach can largely
reduce the search space by introducing prior knowledge in the search process.
Extensive experimental results show that our GPT-NAS method significantly
outperforms seven manually designed neural architectures and thirteen
architectures provided by competing NAS methods. In addition, our ablation
study indicates that the proposed algorithm improves the performance of finely
tuned neural architectures by up to about 12% compared to those without GPT,
further demonstrating its effectiveness in searching neural architectures.

GPT-NAS 是一种基于预训练模型的神经结构搜索算法，通过引入先验知识来大大减少搜索空间，从而优化神经网络架构，并在实验中取得了比其他七个手动设计的神经结构和竞争 NAS 方法提供的 13 个神经结构的好结果，并提高了经过精细调整的神经结构的性能。

GPT-NAS: 采用生成式预训练模型进行神经架构搜索

GPT-NAS: Neural Architecture Search with the Generative Pre-Trained  Model

Deep learning models require extensive architecture design exploration and
hyperparameter optimization to perform well on a given task. The exploration of
the model design space is often made by a human expert, and optimized using a
combination of grid search and search heuristics over a large space of possible
choices. Neural Architecture Search (NAS) is a Reinforcement Learning approach
that has been proposed to automate architecture design. NAS has been
successfully applied to generate Neural Networks that rival the best
human-designed architectures. However, NAS requires sampling, constructing, and
training hundreds to thousands of models to achieve well-performing
architectures. This procedure needs to be executed from scratch for each new
task. The application of NAS to a wide set of tasks currently lacks a way to
transfer generalizable knowledge across tasks. In this paper, we present the
Multitask Neural Model Search (MNMS) controller. Our goal is to learn a
generalizable framework that can condition model construction on successful
model searches for previously seen tasks, thus significantly speeding up the
search for new tasks. We demonstrate that MNMS can conduct an automated
architecture search for multiple tasks simultaneously while still learning
well-performing, specialized models for each task. We then show that
pre-trained MNMS controllers can transfer learning to new tasks. By leveraging
knowledge from previous searches, we find that pre-trained MNMS models start
from a better location in the search space and reduce search time on unseen
tasks, while still discovering models that outperform published human-designed
models.

本文介绍了不同层面的深度学习神经网络结构搜索方法，指出当前手动搜索神经网络结构的困难和繁琐，提出了一种基于强化学习和多任务学习的神经网络结构自动化搜索算法 -- 多任务神经模型搜索算法 (MNMS)，可用于自动化拟合多项任务，其最大的特点在于，采用了迁移学习的思路，将训练模型所获得的知识迁移到下一个任务当中，大幅度节省了搜索时间，提高了模型效果。