Filter level pruning is an effective method to accelerate the inference speed
of deep CNN models. Although numerous pruning algorithms have been proposed,
there are still two open issues. The first problem is how to prune residual
connections. We propose to prune both channels inside and outside the residual
connections via a KL-divergence based criterion. The second issue is pruning
with limited data. We observe an interesting phenomenon: directly pruning on a
small dataset is usually worse than fine-tuning a small model which is pruned
or trained from scratch on the large dataset. Knowledge distillation is an
effective approach to compensate for the weakness of limited data. However, the
logits of a teacher model may be noisy. In order to avoid the influence of
label noise, we propose a label refinement approach to solve this problem.
Experiments have demonstrated the effectiveness of our method (CURL,
Compression Using Residual-connections and Limited-data). CURL significantly
outperforms previous state-of-the-art methods on ImageNet. More importantly,
when pruning on small datasets, CURL achieves comparable or much better
performance than fine-tuning a pretrained small model.

本研究提出了 CURL 方法，通过 KL-divergence 标准对残差连接内外的通道进行修剪，并使用知识蒸馏和标签细化方法解决了有限数据和标签噪声带来的问题，在 ImageNet 数据集上显著优于之前的最先进方法，同时在小数据集上修剪时，与预训练的小模型微调效果相当甚至更好。

具有残差连接和有限数据的神经网络剪枝

Neural Network Pruning with Residual-Connections and Limited-Data

We propose an efficient and unified framework, namely ThiNet, to
simultaneously accelerate and compress CNN models in both training and
inference stages. We focus on the filter level pruning, i.e., the whole filter
would be discarded if it is less important. Our method does not change the
original network structure, thus it can be perfectly supported by any
off-the-shelf deep learning libraries. We formally establish filter pruning as
an optimization problem, and reveal that we need to prune filters based on
statistics information computed from its next layer, not the current layer,
which differentiates ThiNet from existing methods. Experimental results
demonstrate the effectiveness of this strategy, which has advanced the
state-of-the-art. We also show the performance of ThiNet on ILSVRC-12
benchmark. ThiNet achieves 3.31$\times$ FLOPs reduction and 16.63$\times$
compression on VGG-16, with only 0.52$\%$ top-5 accuracy drop. Similar
experiments with ResNet-50 reveal that even for a compact network, ThiNet can
also reduce more than half of the parameters and FLOPs, at the cost of roughly
1$\%$ top-5 accuracy drop. Moreover, the original VGG-16 model can be further
pruned into a very small model with only 5.05MB model size, preserving AlexNet
level accuracy but showing much stronger generalization ability.

ThiNet 是一种快速压缩 CNN 模型的框架，通过统计信息计算从下一层获取关键信息来剪枝，可实现 FLOPs 减少和模型压缩。实验结果表明，即使对于紧凑网络，ThiNet 也能减少一半以上的参数和 FLOPs，牺牲不到 1% 的精度。