Neural architecture search (NAS) automatically finds the best task-specific
neural network topology, outperforming many manual architecture designs.
However, it can be prohibitively expensive as the search requires training
thousands of different networks, while each can last for hours. In this work,
we propose the Graph HyperNetwork (GHN) to amortize the search cost: given an
architecture, it directly generates the weights by running inference on a graph
neural network. GHNs model the topology of an architecture and therefore can
predict network performance more accurately than regular hypernetworks and
premature early stopping. To perform NAS, we randomly sample architectures and
use the validation accuracy of networks with GHN generated weights as the
surrogate search signal. GHNs are fast -- they can search nearly 10 times
faster than other random search methods on CIFAR-10 and ImageNet. GHNs can be
further extended to the anytime prediction setting, where they have found
networks with better speed-accuracy tradeoff than the state-of-the-art manual
designs.

本文提出了一种新的神经网络结构搜索方法 ——Graph HyperNetworks，借助图神经网络来预测网络性能并直接生成权重，从而提高神经网络搜索效率；此外，Graph HyperNetworks 还可在任意预测环境下使用，在 CIFAR-10 和 ImageNet 等数据集上比现有手动设计网络更快且性能更好。

图形超网络用于神经架构搜索

Graph HyperNetworks for Neural Architecture Search

The pioneer deep neural networks (DNNs) have emerged to be deeper or wider
for improving their accuracy in various applications of artificial
intelligence. However, DNNs are often too heavy to deploy in practice, and it
is often required to control their architectures dynamically given computing
resource budget, i.e., anytime prediction. While most existing approaches have
focused on training multiple shallow sub-networks jointly, we study training
thin sub-networks instead. To this end, we first build many inclusive thin
sub-networks (of the same depth) under a minor modification of existing
multi-branch DNNs, and found that they can significantly outperform the
state-of-art dense architecture for anytime prediction. This is remarkable due
to their simplicity and effectiveness, but training many thin sub-networks
jointly faces a new challenge on training complexity. To address the issue, we
also propose a novel DNN architecture by forcing a certain sparsity pattern on
multi-branch network parameters, making them train efficiently for the purpose
of anytime prediction. In our experiments on the ImageNet dataset, its
sub-networks have up to $43.3\%$ smaller sizes (FLOPs) compared to those of the
state-of-art anytime model with respect to the same accuracy. Finally, we also
propose an alternative task under the proposed architecture using a
hierarchical taxonomy, which brings a new angle for anytime prediction.

在本文中，我们提出了许多包容性的全新的 Thin Sub-networks 架构，并通过强制在多分支网络参数上施加某些稀疏模式来训练它们，实现更高效的任意时间预测。我们在 ImageNet 数据集上进行的实验表明，相对于同等准确性的任意时间模型，其子网络可减少高达 43.3％的大小（FLOPs）

基于竖向切片网络的任意时间神经预测

Anytime Neural Prediction via Slicing Networks Vertically

We propose Impatient Deep Neural Networks (DNNs) which deal with dynamic time
budgets during application. They allow for individual budgets given a priori
for each test example and for anytime prediction, i.e., a possible interruption
at multiple stages during inference while still providing output estimates. Our
approach can therefore tackle the computational costs and energy demands of
DNNs in an adaptive manner, a property essential for real-time applications.
Our Impatient DNNs are based on a new general framework of learning dynamic
budget predictors using risk minimization, which can be applied to current DNN
architectures by adding early prediction and additional loss layers. A key
aspect of our method is that all of the intermediate predictors are learned
jointly. In experiments, we evaluate our approach for different budget
distributions, architectures, and datasets. Our results show a significant gain
in expected accuracy compared to common baselines.

本文提出了一种新的深度神经网络模型 ——Impatient Deep Neural Networks，因应动态时间预算且同时支持随时停顿多阶段推断，此方法可实现自适应计算及能源需求，提高实时应用性能，其重要特征为联合学习动态预算预测器。实验结果表明，与常见基线相比，我们的方法在预测精度等方面有较大提升。