Channel pruning can significantly accelerate and compress deep neural
networks. Many channel pruning works utilize structured sparsity regularization
to zero out all the weights in some channels and automatically obtain
structure-sparse network in training stage. However, these methods apply
structured sparsity regularization on each layer separately where the
correlations between consecutive layers are omitted. In this paper, we first
combine one out-channel in current layer and the corresponding in-channel in
next layer as a regularization group, namely out-in-channel. Our proposed
Out-In-Channel Sparsity Regularization (OICSR) considers correlations between
successive layers to further retain predictive power of the compact network.
Training with OICSR thoroughly transfers discriminative features into a
fraction of out-in-channels. Correspondingly, OICSR measures channel importance
based on statistics computed from two consecutive layers, not individual layer.
Finally, a global greedy pruning algorithm is designed to remove redundant
out-in-channels in an iterative way. Our method is comprehensively evaluated
with various CNN architectures including CifarNet, AlexNet, ResNet, DenseNet
and PreActSeNet on CIFAR-10, CIFAR-100 and ImageNet-1K datasets. Notably, on
ImageNet-1K, we reduce 37.2% FLOPs on ResNet-50 while outperforming the
original model by 0.22% top-1 accuracy.

本文提出了一种基于 Out-In-Channel 稀疏正则化（OICSR）的全局贪心利用相关性检测权重，进而实现对多个连续卷积层的稀疏化压缩，并设计了一种全局贪心裁剪算法来进一步压缩卷积神经网络的 FLOPs，同时在准确率方面也取得了优于原始模型的成果。

OICSR：用于紧凑型深度神经网络的通道内外稀疏正则化

OICSR: Out-In-Channel Sparsity Regularization for Compact Deep Neural  Networks

The success of convolutional neural networks (CNNs) in computer vision
applications has been accompanied by a significant increase of computation and
memory costs, which prohibits its usage on resource-limited environments such
as mobile or embedded devices. To this end, the research of CNN compression has
recently become emerging. In this paper, we propose a novel filter pruning
scheme, termed structured sparsity regularization (SSR), to simultaneously
speedup the computation and reduce the memory overhead of CNNs, which can be
well supported by various off-the-shelf deep learning libraries. Concretely,
the proposed scheme incorporates two different regularizers of structured
sparsity into the original objective function of filter pruning, which fully
coordinates the global outputs and local pruning operations to adaptively prune
filters. We further propose an Alternative Updating with Lagrange Multipliers
(AULM) scheme to efficiently solve its optimization. AULM follows the principle
of ADMM and alternates between promoting the structured sparsity of CNNs and
optimizing the recognition loss, which leads to a very efficient solver (2.5x
to the most recent work that directly solves the group sparsity-based
regularization). Moreover, by imposing the structured sparsity, the online
inference is extremely memory-light, since the number of filters and the output
feature maps are simultaneously reduced. The proposed scheme has been deployed
to a variety of state-of-the-art CNN structures including LeNet, AlexNet, VGG,
ResNet and GoogLeNet over different datasets. Quantitative results demonstrate
that the proposed scheme achieves superior performance over the
state-of-the-art methods. We further demonstrate the proposed compression
scheme for the task of transfer learning, including domain adaptation and
object detection, which also show exciting performance gains over the
state-of-the-arts.

本文提出了一种名为结构化稀疏正则化（SSR）的滤波器剪枝方案，它通过两种不同的正则化方法来适应性地裁剪卷积神经网络（CNN）的滤波器，从而提高计算速度和降低内存开销，并通过实验验证了其优异性能。