We propose an algorithm capable of identifying and eliminating irrelevant layers of a neural network during the early stages of training. In contrast to weight or filter-level pruning, layer pruning reduces the harder to parallelize sequential computation of a neural network. We employ a structure using residual connections around nonlinear network sections that allow the flow of information through the network once a nonlinear section is pruned. Our approach is based on variational inference principles using Gaussian scale mixture priors on the neural network weights and allows for substantial cost savings during both training and inference. More specifically, the variational posterior distribution of scalar Bernoulli random variables multiplying a layer weight matrix of its nonlinear sections is learned, similarly to adaptive layer-wise dropout. To overcome challenges of concurrent learning and pruning such as premature pruning and lack of robustness with respect to weight initialization or the size of the starting network, we adopt the "flattening" hyper-prior on the prior parameters. We prove that, as a result of its usage, the solutions of the resulting optimization problem describe deterministic networks with parameters of the posterior distribution at either 0 or 1. We formulate a projected SGD algorithm and prove its convergence to such a solution using stochastic approximation results. In particular, we prove conditions that lead to a layer's weights converging to zero and derive practical pruning conditions from the theoretical results. The proposed algorithm is evaluated on the MNIST, CIFAR-10 and ImageNet datasets and common LeNet, VGG16 and ResNet architectures. The simulations demonstrate that our method achieves state-of the-art performance for layer pruning at reduced computational cost in distinction to competing methods due to the concurrent training and pruning.

我们提出了一种在神经网络早期训练阶段识别和消除不相关层的算法。与权重或滤波器层剪枝相比，层剪枝能够减少神经网络中更难并行计算的顺序计算。我们采用了一种结构，在非线性网络部分周围使用剩余连接，使非线性部分进行剪枝后仍然能够在网络中传递信息。我们的方法基于变分推断原理，在神经网络权重上使用高斯规模混合先验，从而在训练和推理过程中实现大幅成本节省。该算法在MNIST、CIFAR-10和ImageNet数据集以及常见的LeNet、VGG16和ResNet架构上进行了评估，模拟实验结果表明，由于并行训练和剪枝，我们的方法在层剪枝方面以较低的计算成本实现了最先进的性能。

深度神经网络的并发训练和层剪枝