We present in this paper a systematic study on how to morph a well-trained
neural network to a new one so that its network function can be completely
preserved. We define this as \emph{network morphism} in this research. After
morphing a parent network, the child network is expected to inherit the
knowledge from its parent network and also has the potential to continue
growing into a more powerful one with much shortened training time. The first
requirement for this network morphism is its ability to handle diverse morphing
types of networks, including changes of depth, width, kernel size, and even
subnet. To meet this requirement, we first introduce the network morphism
equations, and then develop novel morphing algorithms for all these morphing
types for both classic and convolutional neural networks. The second
requirement for this network morphism is its ability to deal with non-linearity
in a network. We propose a family of parametric-activation functions to
facilitate the morphing of any continuous non-linear activation neurons.
Experimental results on benchmark datasets and typical neural networks
demonstrate the effectiveness of the proposed network morphism scheme.

本文介绍了网络形态学的定义、方程和算法，讨论了如何实现网络形态的改变，提出了参数化激活函数，实验结果表明这种网络形态的改变方案是有效的。