Network pruning is widely used to lighten and accelerate neural network
models. Structured network pruning discards the whole neuron or filter, leading
to accuracy loss. In this work, we propose a novel concept of neuron merging
applicable to both fully connected layers and convolution layers, which
compensates for the information loss due to the pruned neurons/filters. Neuron
merging starts with decomposing the original weights into two matrices/tensors.
One of them becomes the new weights for the current layer, and the other is
what we name a scaling matrix, guiding the combination of neurons. If the
activation function is ReLU, the scaling matrix can be absorbed into the next
layer under certain conditions, compensating for the removed neurons. We also
propose a data-free and inexpensive method to decompose the weights by
utilizing the cosine similarity between neurons. Compared to the pruned model
with the same topology, our merged model better preserves the output feature
map of the original model; thus, it maintains the accuracy after pruning
without fine-tuning. We demonstrate the effectiveness of our approach over
network pruning for various model architectures and datasets. As an example,
for VGG-16 on CIFAR-10, we achieve an accuracy of 93.16% while reducing 64% of
total parameters, without any fine-tuning. The code can be found here:
this https URL

本研究提出了一种新的神经元合并方法，通过分解原始权重矩阵，使用一个被称为缩放矩阵的指导矩阵来引导神经元的组合，以补偿由网络剪枝所导致的信息损失，并在各种模型结构和数据集上证实了本方法的有效性。