We propose ResRep, a novel method for lossless channel pruning (a.k.a. filter
pruning), which slims down a CNN by reducing the width (number of output
channels) of convolutional layers. Inspired by the neurobiology research about
the independence of remembering and forgetting, we propose to re-parameterize a
CNN into the remembering parts and forgetting parts, where the former learn to
maintain the performance and the latter learn to prune. Via training with
regular SGD on the former but a novel update rule with penalty gradients on the
latter, we realize structured sparsity. Then we equivalently merge the
remembering and forgetting parts into the original architecture with narrower
layers. In this sense, ResRep can be viewed as a successful application of
Structural Re-parameterization. Such a methodology distinguishes ResRep from
the traditional learning-based pruning paradigm that applies a penalty on
parameters to produce sparsity, which may suppress the parameters essential for
the remembering. ResRep slims down a standard ResNet-50 with 76.15% accuracy on
ImageNet to a narrower one with only 45% FLOPs and no accuracy drop, which is
the first to achieve lossless pruning with such a high compression ratio. The
code and models are at this https URL

本研究提出了 ResRep 方法，该方法可通过重要参数的记忆和抛弃来实现有结构的稀疏，并且可以在不损失精度的情况下压缩 ResNet-50 模型，实现了高压缩率下的无损剪枝。