Compared to the moderate size of neural network models, structural weight
pruning on the Large-Language Models (LLMs) imposes a novel challenge on the
efficiency of the pruning algorithms, due to the heavy computation/memory
demands of the LLMs. Recent efficient LLM pruning methods typically operate at
the post-training phase without the expensive weight finetuning, however, their
pruning criteria often rely on heuristically designed metrics, potentially
leading to suboptimal performance. We instead propose a novel
optimization-based structural pruning that learns the pruning masks in a
probabilistic space directly by optimizing the loss of the pruned model. To
preserve the efficiency, our method 1) works at post-training phase} and 2)
eliminates the back-propagation through the LLM per se during the optimization
(i.e., only requires the forward pass of the LLM). We achieve this by learning
an underlying Bernoulli distribution to sample binary pruning masks, where we
decouple the Bernoulli parameters from the LLM loss, thus facilitating an
efficient optimization via a policy gradient estimator without
back-propagation. As a result, our method is able to 1) operate at structural
granularities of channels, heads, and layers, 2) support global and
heterogeneous pruning (i.e., our method automatically determines different
redundancy for different layers), and 3) optionally use a metric-based method
as initialization (of our Bernoulli distributions). Extensive experiments on
LLaMA, LLaMA-2, and Vicuna using the C4 and WikiText2 datasets demonstrate that
our method operates for 2.7 hours with around 35GB memory for the 13B models on
a single A100 GPU, and our pruned models outperform the state-of-the-arts
w.r.t. perplexity. Codes will be released.

基于优化的结构剪枝方法通过在概率空间中学习剪枝掩码，通过前向传递和策略梯度估计器进行高效优化，实现对大型语言模型的剪枝，并在复杂性和效果方面超越现有方法。

大规模语言模型的优化结构裁剪方法

Optimization-based Structural Pruning for Large Language Models without  Back-Propagation

It is widely acknowledged that large and sparse models have higher accuracy
than small and dense models under the same model size constraints. This
motivates us to train a large model and then remove its redundant neurons or
weights by pruning. Most existing works pruned the networks in a deterministic
way, the performance of which solely depends on a single pruning criterion and
thus lacks variety. Instead, in this paper, we propose a model pruning strategy
that first generates several pruning masks in a designed random way.
Subsequently, along with an effective mask-selection rule, the optimal mask is
chosen from the pool of mask candidates. To further enhance efficiency, we
introduce an early mask evaluation strategy, mitigating the overhead associated
with training multiple masks. Our extensive experiments demonstrate that this
approach achieves state-of-the-art performance across eight datasets from GLUE,
particularly excelling at high levels of sparsity.

在同等模型大小约束下，大而稀疏的模型普遍比小而密集的模型具有更高的准确性。因此，本文提出了一种模型修剪策略，首先以设计的随机方式生成多个修剪掩码，然后通过一个有效的掩码选择规则从候选掩码组中选择最优掩码。为了进一步提高效率，引入了一种早期掩码评估策略，减少了训练多个掩码的开销。在 GLUE 数据集的八个数据集上进行了广泛的实验证明，该方法在高稀疏水平下实现了最先进的性能。

突破确定性障碍：随机修剪掩码的生成和选择

Breaking through Deterministic Barriers: Randomized Pruning Mask  Generation and Selection

Debiasing language models from unwanted behaviors in Natural Language
Understanding tasks is a topic with rapidly increasing interest in the NLP
community. Spurious statistical correlations in the data allow models to
perform shortcuts and avoid uncovering more advanced and desirable linguistic
features. A multitude of effective debiasing approaches has been proposed, but
flexibility remains a major issue. For the most part, models must be retrained
to find a new set of weights with debiased behavior. We propose a new debiasing
method in which we identify debiased pruning masks that can be applied to a
finetuned model. This enables the selective and conditional application of
debiasing behaviors. We assume that bias is caused by a certain subset of
weights in the network; our method is, in essence, a mask search to identify
and remove biased weights. Our masks show equivalent or superior performance to
the standard counterparts, while offering important benefits. Pruning masks can
be stored with high efficiency in memory, and it becomes possible to switch
among several debiasing behaviors (or revert back to the original biased model)
at inference time. Finally, it opens the doors to further research on how
biases are acquired by studying the generated masks. For example, we observed
that the early layers and attention heads were pruned more aggressively,
possibly hinting towards the location in which biases may be encoded.

介绍一种新的语言模型去偏置化的方法，使用剪枝口罩来确定模型去偏置化行为，不需要重新训练模型，可以高效地存储口罩并在推断时切换到多种去偏置行为。