We propose Stochastic Neural Architecture Search (SNAS), an economical
end-to-end solution to Neural Architecture Search (NAS) that trains neural
operation parameters and architecture distribution parameters in same round of
back-propagation, while maintaining the completeness and differentiability of
the NAS pipeline. In this work, NAS is reformulated as an optimization problem
on parameters of a joint distribution for the search space in a cell. To
leverage the gradient information in generic differentiable loss for
architecture search, a novel search gradient is proposed. We prove that this
search gradient optimizes the same objective as reinforcement-learning-based
NAS, but assigns credits to structural decisions more efficiently. This credit
assignment is further augmented with locally decomposable reward to enforce a
resource-efficient constraint. In experiments on CIFAR-10, SNAS takes less
epochs to find a cell architecture with state-of-the-art accuracy than
non-differentiable evolution-based and reinforcement-learning-based NAS, which
is also transferable to ImageNet. It is also shown that child networks of SNAS
can maintain the validation accuracy in searching, with which attention-based
NAS requires parameter retraining to compete, exhibiting potentials to stride
towards efficient NAS on big datasets. We have released our implementation at
this https URL

提出了随机神经架构搜索 (SNAS)，作为一种经济高效的神经架构搜索方案，它在同一次反向传播中训练神经操作参数和架构分布参数，同时保持 NAS 管道的完整性和可区分性。通过对单元搜索空间的联合分布的参数进行优化来重构 NAS，提出一种全新的搜索梯度，优化效果比基于强化学习的 NAS 更高，同时加入局部可分解奖励，以强制实施资源有效约束，最终在 CIFAR-10 数据集上得到了业界领先的准确性。