We propose a novel method to merge convolutional neural-nets for the
inference stage. Given two well-trained networks that may have different
architectures that handle different tasks, our method aligns the layers of the
original networks and merges them into a unified model by sharing the
representative codes of weights. The shared weights are further re-trained to
fine-tune the performance of the merged model. The proposed method effectively
produces a compact model that may run original tasks simultaneously on
resource-limited devices. As it preserves the general architectures and
leverages the co-used weights of well-trained networks, a substantial training
overhead can be reduced to shorten the system development time. Experimental
results demonstrate a satisfactory performance and validate the effectiveness
of the method.

本文提出了一种新的卷积神经网络合并方法，在共享权重的基础上，将训练好的两个具有不同架构的网络在推断阶段融合为一个统一的模型，以适应资源有限的设备，实验结果表明此方法有效。

深度神经网络的推理阶段统一和合并

Unifying and Merging Well-trained Deep Neural Networks for Inference  Stage

We analyze the loss landscape and expressiveness of practical deep
convolutional neural networks (CNNs) with shared weights and max pooling
layers. We show that such CNNs produce linearly independent features at a
"wide" layer which has more neurons than the number of training samples. This
condition holds e.g. for the VGG network. Furthermore, we provide for such wide
CNNs necessary and sufficient conditions for global minima with zero training
error. For the case where the wide layer is followed by a fully connected layer
we show that almost every critical point of the empirical loss is a global
minimum with zero training error. Our analysis suggests that both depth and
width are very important in deep learning. While depth brings more
representational power and allows the network to learn high level features,
width smoothes the optimization landscape of the loss function in the sense
that a sufficiently wide network has a well-behaved loss surface with almost no
bad local minima.

研究了使用共享权重和最大池化层的实用深度卷积神经网络的损失景观和表现力，发现这些 CNN 在宽层处可以产生线性独立特征，研究了获得零训练误差的全局最小值所需的必要和充分条件，分析表明，深度和宽度都很重要，宽度可以使损失函数具有很好的性质。