Techniques for automatically designing deep neural network architectures such
as reinforcement learning based approaches have recently shown promising
results. However, their success is based on vast computational resources (e.g.
hundreds of GPUs), making them difficult to be widely used. A noticeable
limitation is that they still design and train each network from scratch during
the exploration of the architecture space, which is highly inefficient. In this
paper, we propose a new framework toward efficient architecture search by
exploring the architecture space based on the current network and reusing its
weights. We employ a reinforcement learning agent as the meta-controller, whose
action is to grow the network depth or layer width with function-preserving
transformations. As such, the previously validated networks can be reused for
further exploration, thus saves a large amount of computational cost. We apply
our method to explore the architecture space of the plain convolutional neural
networks (no skip-connections, branching etc.) on image benchmark datasets
(CIFAR-10, SVHN) with restricted computational resources (5 GPUs). Our method
can design highly competitive networks that outperform existing networks using
the same design scheme. On CIFAR-10, our model without skip-connections
achieves 4.23\% test error rate, exceeding a vast majority of modern
architectures and approaching DenseNet. Furthermore, by applying our method to
explore the DenseNet architecture space, we are able to achieve more accurate
networks with fewer parameters.

本文提出了一种基于深度神经网络的自动化设计架构的新框架，该框架使用强化学习代理作为元控制器，通过功能保持变换在当前网络的基础上，尝试增加网络深度或层宽。在 CIFAR-10 和 SVHN 基准数据集上的测试表明，该方法在受限的计算资源下可以设计出具有高竞争力的网络，并在不使用跨层连接的条件下实现了 4.23％的测试误差率。

通过网络转换实现高效架构搜索

Efficient Architecture Search by Network Transformation

We introduce techniques for rapidly transferring the information stored in
one neural net into another neural net. The main purpose is to accelerate the
training of a significantly larger neural net. During real-world workflows, one
often trains very many different neural networks during the experimentation and
design process. This is a wasteful process in which each new model is trained
from scratch. Our Net2Net technique accelerates the experimentation process by
instantaneously transferring the knowledge from a previous network to each new
deeper or wider network. Our techniques are based on the concept of
function-preserving transformations between neural network specifications. This
differs from previous approaches to pre-training that altered the function
represented by a neural net when adding layers to it. Using our knowledge
transfer mechanism to add depth to Inception modules, we demonstrate a new
state of the art accuracy rating on the ImageNet dataset.

介绍了快速将一个神经网络中存储的信息传输到另一个神经网络中的技术，并演示了将 Net2Net 技术用于 Inception 模块增加深度时，对 ImageNet 数据集取得了新的最先进的准确度评分。