As a popular channel pruning method for convolutional neural networks (CNNs),
network slimming (NS) has a three-stage process: (1) it trains a CNN with
$\ell_1$ regularization applied to the scaling factors of the batch
normalization layers; (2) it removes channels whose scaling factors are below a
chosen threshold; and (3) it retrains the pruned model to recover the original
accuracy. This time-consuming, three-step process is a result of using
subgradient descent to train CNNs. Because subgradient descent does not exactly
train CNNs towards sparse, accurate structures, the latter two steps are
necessary. Moreover, subgradient descent does not have any convergence
guarantee. Therefore, we develop an alternative algorithm called proximal NS.
Our proposed algorithm trains CNNs towards sparse, accurate structures, so
identifying a scaling factor threshold is unnecessary and fine tuning the
pruned CNNs is optional. Using Kurdyka-{\L}ojasiewicz assumptions, we establish
global convergence of proximal NS. Lastly, we validate the efficacy of the
proposed algorithm on VGGNet, DenseNet and ResNet on CIFAR 10/100. Our
experiments demonstrate that after one round of training, proximal NS yields a
CNN with competitive accuracy and compression.

研究提出一种叫做 proximal NS 的算法，可以训练卷积神经网络达到精炼、准确的结构，无需识别阈值并且压缩效果较好。实验中使用该算法在 CIFAR 10/100 数据集上验证了其有效性。

网络瘦身的近端算法

A Proximal Algorithm for Network Slimming

The deployment of deep convolutional neural networks (CNNs) in many real
world applications is largely hindered by their high computational cost. In
this paper, we propose a novel learning scheme for CNNs to simultaneously 1)
reduce the model size; 2) decrease the run-time memory footprint; and 3) lower
the number of computing operations, without compromising accuracy. This is
achieved by enforcing channel-level sparsity in the network in a simple but
effective way. Different from many existing approaches, the proposed method
directly applies to modern CNN architectures, introduces minimum overhead to
the training process, and requires no special software/hardware accelerators
for the resulting models. We call our approach network slimming, which takes
wide and large networks as input models, but during training insignificant
channels are automatically identified and pruned afterwards, yielding thin and
compact models with comparable accuracy. We empirically demonstrate the
effectiveness of our approach with several state-of-the-art CNN models,
including VGGNet, ResNet and DenseNet, on various image classification
datasets. For VGGNet, a multi-pass version of network slimming gives a 20x
reduction in model size and a 5x reduction in computing operations.

本文提出了一种名为 “网络瘦身” 的新颖学习方案，可以通过引入通道级稀疏性，简化深度卷积神经网络，降低了其计算成本，减小了运行时内存占用，同时不影响准确度。我们进行了多组实验，证明了该方案在各种图像分类数据集上具有较好的效果。