Given a pre-trained language model, how can we efficiently compress it
without retraining? Retraining-free structured pruning algorithms are crucial
in pre-trained language model compression due to their significantly reduced
pruning cost and capability to prune large language models. However, existing
retraining-free algorithms encounter severe accuracy degradation, as they fail
to preserve the useful knowledge of pre-trained models. In this paper, we
propose K-pruning (Knowledge-preserving pruning), an accurate retraining-free
structured pruning algorithm for pre-trained language models. K-pruning
identifies and prunes attention heads and neurons deemed to be superfluous,
based on the amount of their inherent knowledge. K-pruning applies an iterative
process of pruning followed by knowledge reconstruction for each sub-layer to
preserve the knowledge of the pre-trained models. Consequently, K-pruning shows
up to 58.02%p higher F1 score than existing retraining-free pruning algorithms
under a high compression rate of 80% on the SQuAD benchmark.

提出了一种名为 K-pruning (Knowledge-preserving pruning) 的准确的无需重新训练的结构化修剪算法，用于预训练的语言模型压缩，并在 SQuAD 基准测试中展示了比现有的无需重新训练的修剪算法高达 58.02% p 的 F1 分数的优势。

无需重新训练的预训练语言模型知识保留剪枝

Knowledge-preserving Pruning for Pre-trained Language Models without  Retraining

Network pruning is widely used for reducing the heavy inference cost of deep
models in low-resource settings. A typical pruning algorithm is a three-stage
pipeline, i.e., training (a large model), pruning and fine-tuning. During
pruning, according to a certain criterion, redundant weights are pruned and
important weights are kept to best preserve the accuracy. In this work, we make
several surprising observations which contradict common beliefs. For all
state-of-the-art structured pruning algorithms we examined, fine-tuning a
pruned model only gives comparable or worse performance than training that
model with randomly initialized weights. For pruning algorithms which assume a
predefined target network architecture, one can get rid of the full pipeline
and directly train the target network from scratch. Our observations are
consistent for multiple network architectures, datasets, and tasks, which imply
that: 1) training a large, over-parameterized model is often not necessary to
obtain an efficient final model, 2) learned "important" weights of the large
model are typically not useful for the small pruned model, 3) the pruned
architecture itself, rather than a set of inherited "important" weights, is
more crucial to the efficiency in the final model, which suggests that in some
cases pruning can be useful as an architecture search paradigm. Our results
suggest the need for more careful baseline evaluations in future research on
structured pruning methods. We also compare with the "Lottery Ticket
Hypothesis" (Frankle & Carbin 2019), and find that with optimal learning rate,
the "winning ticket" initialization as used in Frankle & Carbin (2019) does not
bring improvement over random initialization.

本研究发现，在针对低资源设置中减少深度模型推理成本的网络修剪过程中，训练大模型通常不是获得高效终端模型的必要条件，学到的 “重要” 权重通常对小模型没有用，修剪的架构本身比继承的 “重要” 权重更重要，并且此方法可作为架构搜索范式。本文还比较了 “Lottery Ticket Hypothesis”，发现在最佳学习率下，与随机初始化相比，其 “获胜券” 初始化并未带来提高。