Neural Architecture Search (NAS) is an exciting new field which promises to
be as much as a game-changer as Convolutional Neural Networks were in 2012.
Despite many great works leading to substantial improvements on a variety of
tasks, comparison between different methods is still very much an open issue.
While most algorithms are tested on the same datasets, there is no shared
experimental protocol followed by all. As such, and due to the under-use of
ablation studies, there is a lack of clarity regarding why certain methods are
more effective than others. Our first contribution is a benchmark of $8$ NAS
methods on $5$ datasets. To overcome the hurdle of comparing methods with
different search spaces, we propose using a method's relative improvement over
the randomly sampled average architecture, which effectively removes advantages
arising from expertly engineered search spaces or training protocols.
Surprisingly, we find that many NAS techniques struggle to significantly beat
the average architecture baseline. We perform further experiments with the
commonly used DARTS search space in order to understand the contribution of
each component in the NAS pipeline. These experiments highlight that: (i) the
use of tricks in the evaluation protocol has a predominant impact on the
reported performance of architectures; (ii) the cell-based search space has a
very narrow accuracy range, such that the seed has a considerable impact on
architecture rankings; (iii) the hand-designed macro-structure (cells) is more
important than the searched micro-structure (operations); and (iv) the
depth-gap is a real phenomenon, evidenced by the change in rankings between $8$
and $20$ cell architectures. To conclude, we suggest best practices, that we
hope will prove useful for the community and help mitigate current NAS
pitfalls. The code used is available at
this https URL

在本文中，我们提出了用于比较 NAS 方法的基准测试，并对常用的 DARTS search space 进行了进一步的实验。我们发现，手工设计的宏观结构比搜索到的微观结构更重要，并且深度间隔是一个真实的现象，最后我们提出了最佳实践以帮助缓解当前的 NAS 缺陷。