Neural architecture search (NAS) has attracted increasing attentions in both
academia and industry. In the early age, researchers mostly applied individual
search methods which sample and evaluate the candidate architectures separately
and thus incur heavy computational overheads. To alleviate the burden,
weight-sharing methods were proposed in which exponentially many architectures
share weights in the same super-network, and the costly training procedure is
performed only once. These methods, though being much faster, often suffer the
issue of instability. This paper provides a literature review on NAS, in
particular the weight-sharing methods, and points out that the major challenge
comes from the optimization gap between the super-network and the
sub-architectures. From this perspective, we summarize existing approaches into
several categories according to their efforts in bridging the gap, and analyze
both advantages and disadvantages of these methodologies. Finally, we share our
opinions on the future directions of NAS and AutoML. Due to the expertise of
the authors, this paper mainly focuses on the application of NAS to computer
vision problems and may bias towards the work in our group.

本文介绍了神经体系结构搜索中的共享权重方法，并提出了其最大的挑战在于超网络与子体系结构之间的优化差距。同时，根据不同的方法来弥合这个差距，分析了这些方法的优劣。最后，就 NAS 和 AutoML 的未来方向提出了作者的观点。

共享权重神经架构搜索：缩小优化差距的战斗

Weight-Sharing Neural Architecture Search: A Battle to Shrink the  Optimization Gap

DARTS is a popular algorithm for neural architecture search (NAS). Despite
its great advantage in search efficiency, DARTS often suffers weak stability,
which reflects in the large variation among individual trials as well as the
sensitivity to the hyper-parameters of the search process. This paper owes such
instability to an optimization gap between the super-network and its
sub-networks, namely, improving the validation accuracy of the super-network
does not necessarily lead to a higher expectation on the performance of the
sampled sub-networks. Then, we point out that the gap is due to the inaccurate
estimation of the architectural gradients, based on which we propose an amended
estimation method. Mathematically, our method guarantees a bounded error from
the true gradients while the original estimation does not. Our approach bridges
the gap from two aspects, namely, amending the estimation on the architectural
gradients, and unifying the hyper-parameter settings in the search and
re-training stages. Experiments on CIFAR10 and ImageNet demonstrate that our
approach largely improves search stability and, more importantly, enables
DARTS-based approaches to explore much larger search spaces that have not been
investigated before.

该文提出 DARTS 算法在神经架构搜索中的优化问题，并提出一种修正的架构梯度估计方法来桥接优化差距，实验结果表明该方法有效提高了搜索稳定性和搜索空间的探索能力。