Recently Neural Architecture Search (NAS) has aroused great interest in both
academia and industry, however it remains challenging because of its huge and
non-continuous search space. Instead of applying evolutionary algorithm or
reinforcement learning as previous works, this paper proposes a Direct Sparse
Optimization NAS (DSO-NAS) method. In DSO-NAS, we provide a novel model pruning
view to NAS problem. In specific, we start from a completely connected block,
and then introduce scaling factors to scale the information flow between
operations. Next, we impose sparse regularizations to prune useless connections
in the architecture. Lastly, we derive an efficient and theoretically sound
optimization method to solve it. Our method enjoys both advantages of
differentiability and efficiency, therefore can be directly applied to large
datasets like ImageNet. Particularly, On CIFAR-10 dataset, DSO-NAS achieves an
average test error 2.84\%, while on the ImageNet dataset DSO-NAS achieves
25.4\% test error under 600M FLOPs with 8 GPUs in 18 hours.

该论文提出了一种名为 DSO-NAS 的神经架构搜索方法，采用基于稀疏正则化的模型修剪视角，利用逐步引入稀疏网络的方法来解决搜索空间庞大且不连续的问题，在 CIFAR-10 数据集上达到了 2.84％的平均测试误差，在 ImageNet 数据集上，仅使用 8 块 GPU 600M FLOPs 进行 18 小时的训练可以达到 25.4％的测试误差。