Developing efficient and principled neural architecture optimization methods
is a critical challenge of modern deep learning. Recently, Liu et al.[19]
proposed a splitting steepest descent (S2D) method that jointly optimizes the
neural parameters and architectures based on progressively growing network
structures by splitting neurons into multiple copies in a steepest descent
fashion. However, S2D suffers from a local optimality issue when all the
neurons become "splitting stable", a concept akin to local stability in
parametric optimization. In this work, we develop a significant and surprising
extension of the splitting descent framework that addresses the local
optimality issue. The idea is to observe that the original S2D is unnecessarily
restricted to splitting neurons into positive weighted copies. By simply
allowing both positive and negative weights during splitting, we can eliminate
the appearance of splitting stability in S2D and hence escape the local optima
to obtain better performance. By incorporating signed splittings, we
significantly extend the optimization power of splitting steepest descent both
theoretically and empirically. We verify our method on various challenging
benchmarks such as CIFAR-100, ImageNet and ModelNet40, on which we outperform
S2D and other advanced methods on learning accurate and energy-efficient neural
networks.

本研究提出通过在当前的神经网络结构中加入正负权重拆分来消除局部最优问题，进而增强神经结构优化效果的方法，应用于 CIFAR-100，ImageNet，ModelNet40 等挑战性数据集上表现优异并超过了同类型算法