Sparsification-based pruning has been an important category in model
compression. Existing methods commonly set sparsity-inducing penalty terms to
suppress the importance of dropped weights, which is regarded as the suppressed
sparsification paradigm. However, this paradigm inactivates the dropped parts
of networks causing capacity damage before pruning, thereby leading to
performance degradation. To alleviate this issue, we first study and reveal the
relative sparsity effect in emerging stimulative training and then propose a
structured pruning framework, named STP, based on an enhanced sparsification
paradigm which maintains the magnitude of dropped weights and enhances the
expressivity of kept weights by self-distillation. Besides, to find an optimal
architecture for the pruned network, we propose a multi-dimension architecture
space and a knowledge distillation-guided exploration strategy. To reduce the
huge capacity gap of distillation, we propose a subnet mutating expansion
technique. Extensive experiments on various benchmarks indicate the
effectiveness of STP. Specifically, without fine-tuning, our method
consistently achieves superior performance at different budgets, especially
under extremely aggressive pruning scenarios, e.g., remaining 95.11% Top-1
accuracy (72.43% in 76.15%) while reducing 85% FLOPs for ResNet-50 on ImageNet.
Codes will be released soon.

基于稀疏化剪枝的研究中，我们提出了一种增强稀疏化范式的结构化剪枝框架 (STP)，通过自蒸馏的方式维持被剪枝权重的大小并增强保留权重的表现力。此外，为了找到最优的剪枝网络架构，我们采用了多维架构空间和知识蒸馏引导的探索策略，同时使用子网变异扩展技术来减小蒸馏的容量差距。大量实验证明了 STP 的有效性，特别是在极度激进的剪枝情况下，例如在 ImageNet 上对 ResNet-50 进行剪枝，保持 95.11% 的 Top-1 准确率（从 76.15% 减少 85% 的浮点操作）。