Neural Architecture Search (NAS) has been explosively studied to automate the
discovery of top-performer neural networks. Current works require heavy
training of supernet or intensive architecture evaluations, thus suffering from
heavy resource consumption and often incurring search bias due to truncated
training or approximations. Can we select the best neural architectures without
involving any training and eliminate a drastic portion of the search cost? We
provide an affirmative answer, by proposing a novel framework called
training-free neural architecture search (TE-NAS). TE-NAS ranks architectures
by analyzing the spectrum of the neural tangent kernel (NTK) and the number of
linear regions in the input space. Both are motivated by recent theory advances
in deep networks and can be computed without any training and any label. We
show that: (1) these two measurements imply the trainability and expressivity
of a neural network; (2) they strongly correlate with the network's test
accuracy. Further on, we design a pruning-based NAS mechanism to achieve a more
flexible and superior trade-off between the trainability and expressivity
during the search. In NAS-Bench-201 and DARTS search spaces, TE-NAS completes
high-quality search but only costs 0.5 and 4 GPU hours with one 1080Ti on
CIFAR-10 and ImageNet, respectively. We hope our work inspires more attempts in
bridging the theoretical findings of deep networks and practical impacts in
real NAS applications. Code is available at:
this https URL

利用神经切向核和输入空间中的线性区域数来排名结构，通过 TE-NAS 框架进行训练 - free 的神经体系结构搜索，使用基于剪枝的 NAS 机制进行灵活且优越的权衡，实现在 CIFAR-10 和 ImageNet 上只花费 0.5 和 4 GPU 小时完成的高质量搜索。