Differentiable architecture search (DARTS) marks a milestone in Neural
Architecture Search (NAS), boasting simplicity and small search costs. However,
DARTS still suffers from frequent performance collapse, which happens when some
operations, such as skip connections, zeroes and poolings, dominate the
architecture. In this paper, we are the first to point out that the phenomenon
is attributed to bi-level optimization. We propose Single-DARTS which merely
uses single-level optimization, updating network weights and architecture
parameters simultaneously with the same data batch. Even single-level
optimization has been previously attempted, no literature provides a systematic
explanation on this essential point. Replacing the bi-level optimization,
Single-DARTS obviously alleviates performance collapse as well as enhances the
stability of architecture search. Experiment results show that Single-DARTS
achieves state-of-the-art performance on mainstream search spaces. For
instance, on NAS-Benchmark-201, the searched architectures are nearly optimal
ones. We also validate that the single-level optimization framework is much
more stable than the bi-level one. We hope that this simple yet effective
method will give some insights on differential architecture search. The code is
available at this https URL

本文提出 Single-DARTS，通过使用单级优化替换双级优化，显著减轻了性能坍塌，提高了体系结构搜索的稳定性，并在主流搜索空间上实现了最先进的性能。