Traditional (unstructured) pruning methods for a Transformer model focus on regularizing the individual weights by penalizing them toward zero. In this work, we explore spectral-normalized identity priors (SNIP), a structured pruning approach that penalizes an entire residual module in a Transformer model toward an identity mapping. Our method identifies and discards unimportant non-linear mappings in the residual connections by applying a thresholding operator on the function norm. It is applicable to any structured module, including a single attention head, an entire attention block, or a feed-forward subnetwork. Furthermore, we introduce spectral normalization to stabilize the distribution of the post-activation values of the Transformer layers, further improving the pruning effectiveness of the proposed methodology. We conduct experiments with BERT on 5 GLUE benchmark tasks to demonstrate that SNIP achieves effective pruning results while maintaining comparable performance. Specifically, we improve the performance over the state-of-the-art by 0.5 to 1.0% on average at 50% compression ratio.

本文介绍了一种基于 SNIP 的结构化剪枝方法，通过对 Transformer 模型中的整个残差模块进行惩罚，识别并丢弃不重要的非线性映射，并引入谱归一化来稳定 Transformer 层的后激活值分布以进一步改善剪枝效果。实验结果表明，SNIP 在保持可比性能的同时实现了有效的剪枝结果，在 50% 的压缩比下将基准任务的性能提高了 0.5 到 1.0%。

通过谱归一化身份先验对Transformer模型中的冗余映射进行修剪