Neural network compression techniques, such as knowledge distillation (KD)
and network pruning, have received increasing attention. Recent work `Prune,
then Distill' reveals that a pruned student-friendly teacher network can
benefit the performance of KD. However, the conventional teacher-student
pipeline, which entails cumbersome pre-training of the teacher and complicated
compression steps, makes pruning with KD less efficient. In addition to
compressing models, recent compression techniques also emphasize the aspect of
efficiency. Early pruning demands significantly less computational cost in
comparison to the conventional pruning methods as it does not require a large
pre-trained model. Likewise, a special case of KD, known as self-distillation
(SD), is more efficient since it requires no pre-training or student-teacher
pair selection. This inspires us to collaborate early pruning with SD for
efficient model compression. In this work, we propose the framework named Early
Pruning with Self-Distillation (EPSD), which identifies and preserves
distillable weights in early pruning for a given SD task. EPSD efficiently
combines early pruning and self-distillation in a two-step process, maintaining
the pruned network's trainability for compression. Instead of a simple
combination of pruning and SD, EPSD enables the pruned network to favor SD by
keeping more distillable weights before training to ensure better distillation
of the pruned network. We demonstrated that EPSD improves the training of
pruned networks, supported by visual and quantitative analyses. Our evaluation
covered diverse benchmarks (CIFAR-10/100, Tiny-ImageNet, full ImageNet,
CUB-200-2011, and Pascal VOC), with EPSD outperforming advanced pruning and SD
techniques.

提出了一种名为早期修剪与自我蒸馏（EPSD）的框架，它在给定自我蒸馏任务的早期修剪中识别和保留可蒸馏的权重，通过两步骤的过程高效地将早期修剪和自我蒸馏相结合，以保持修剪网络的可训练性进行模型压缩。