Neural architecture search (NAS) has shown encouraging results in automating
the architecture design. Recently, DARTS relaxes the search process with a
differentiable formulation that leverages weight-sharing and SGD where all
candidate operations are trained simultaneously. Our empirical results show
that such procedure results in the co-adaption problem and Matthew Effect:
operations with fewer parameters would be trained maturely earlier. This causes
two problems: firstly, the operations with more parameters may never have the
chance to express the desired function since those with less have already done
the job; secondly, the system will punish those underperforming operations by
lowering their architecture parameter, and they will get smaller loss
gradients, which causes the Matthew Effect. In this paper, we systematically
study these problems and propose a novel grouped operation dropout algorithm
named DropNAS to fix the problems with DARTS. Extensive experiments demonstrate
that DropNAS solves the above issues and achieves promising performance.
Specifically, DropNAS achieves 2.26% test error on CIFAR-10, 16.39% on
CIFAR-100 and 23.4% on ImageNet (with the same training hyperparameters as
DARTS for a fair comparison). It is also observed that DropNAS is robust across
variants of the DARTS search space. Code is available at
this https URL

本文研究了 DARTS 算法在神经结构搜索中存在的共适应问题和 Matthew 效应，并提出了一种名为 DropNAS 的新方法来解决这些问题，在 CIFAR-10、CIFAR-100 和 ImageNet 数据集上均获得了良好的表现。