Finding the best neural network architecture requires significant time,
resources, and human expertise. These challenges are partially addressed by
neural architecture search (NAS) which is able to find the best convolutional
layer or cell that is then used as a building block for the network. However,
once a good building block is found, manual design is still required to
assemble the final architecture as a combination of multiple blocks under a
predefined parameter budget constraint. A common solution is to stack these
blocks into a single tower and adjust the width and depth to fill the parameter
budget. However, these single tower architectures may not be optimal. Instead,
in this paper we present the AdaNAS algorithm, that uses ensemble techniques to
compose a neural network as an ensemble of smaller networks automatically.
Additionally, we introduce a novel technique based on knowledge distillation to
iteratively train the smaller networks using the previous ensemble as a
teacher. Our experiments demonstrate that ensembles of networks improve
accuracy upon a single neural network while keeping the same number of
parameters. Our models achieve comparable results with the state-of-the-art on
CIFAR-10 and sets a new state-of-the-art on CIFAR-100.

本文提出了 AdaNAS 算法，利用集成技术将神经网络组合成一组小型神经网络集合，并采用基于知识蒸馏的新技术，迭代地训练小型神经网络，取得了 CIFAR-10 数据集中可比较的结果和 CIFAR-100 数据集中的最佳结果。