Neural architecture search (NAS) has attracted increasing attentions in both
academia and industry. In the early age, researchers mostly applied individual
search methods which sample and evaluate the candidate architectures separately
and thus incur heavy computational overheads. To alleviate the burden,
weight-sharing methods were proposed in which exponentially many architectures
share weights in the same super-network, and the costly training procedure is
performed only once. These methods, though being much faster, often suffer the
issue of instability. This paper provides a literature review on NAS, in
particular the weight-sharing methods, and points out that the major challenge
comes from the optimization gap between the super-network and the
sub-architectures. From this perspective, we summarize existing approaches into
several categories according to their efforts in bridging the gap, and analyze
both advantages and disadvantages of these methodologies. Finally, we share our
opinions on the future directions of NAS and AutoML. Due to the expertise of
the authors, this paper mainly focuses on the application of NAS to computer
vision problems and may bias towards the work in our group.

本文介绍了神经体系结构搜索中的共享权重方法，并提出了其最大的挑战在于超网络与子体系结构之间的优化差距。同时，根据不同的方法来弥合这个差距，分析了这些方法的优劣。最后，就 NAS 和 AutoML 的未来方向提出了作者的观点。

共享权重神经架构搜索：缩小优化差距的战斗

Weight-Sharing Neural Architecture Search: A Battle to Shrink the  Optimization Gap

The search cost of neural architecture search (NAS) has been largely reduced
by weight-sharing methods. These methods optimize a super-network with all
possible edges and operations, and determine the optimal sub-network by
discretization, \textit{i.e.}, pruning off weak candidates. The discretization
process, performed on either operations or edges, incurs significant inaccuracy
and thus the quality of the final architecture is not guaranteed. This paper
presents discretization-aware architecture search (DA\textsuperscript{2}S),
with the core idea being adding a loss term to push the super-network towards
the configuration of desired topology, so that the accuracy loss brought by
discretization is largely alleviated. Experiments on standard image
classification benchmarks demonstrate the superiority of our approach, in
particular, under imbalanced target network configurations that were not
studied before.

本篇论文提出了一种基于唯一拓扑目标的离线训练过程，以大大减少离散化过程中的不准确性，并针对未曾研究的非平衡目标网络配置进行了实验证明其优越性。

离散感知的架构搜索

Discretization-Aware Architecture Search

Neural architecture search has attracted wide attentions in both academia and
industry. To accelerate it, researchers proposed weight-sharing methods which
first train a super-network to reuse computation among different operators,
from which exponentially many sub-networks can be sampled and efficiently
evaluated. These methods enjoy great advantages in terms of computational
costs, but the sampled sub-networks are not guaranteed to be estimated
precisely unless an individual training process is taken. This paper owes such
inaccuracy to the inevitable mismatch between assembled network layers, so that
there is a random error term added to each estimation. We alleviate this issue
by training a graph convolutional network to fit the performance of sampled
sub-networks so that the impact of random errors becomes minimal. With this
strategy, we achieve a higher rank correlation coefficient in the selected set
of candidates, which consequently leads to better performance of the final
architecture. In addition, our approach also enjoys the flexibility of being
used under different hardware constraints, since the graph convolutional
network has provided an efficient lookup table of the performance of
architectures in the entire search space.

本文针对神经架构搜索中子网络精度不足的问题，提出一种基于图卷积网络的训练策略，提升搜索性能，实现更高的排名相关系数。同时，该方法还具有适用于多种硬件限制的灵活性。