Few-shot gradient methods have been extensively utilized in existing model
pruning methods, where the model weights are regarded as static values and the
effects of potential weight perturbations are not considered. However, the
widely used large language models (LLMs) have several billion model parameters,
which could increase the fragility of few-shot gradient pruning. In this work,
we experimentally show that one-shot gradient pruning algorithms could lead to
unstable results under perturbations to model weights. And the minor error of
switching between data formats bfloat16 and float16 could result in drastically
different outcomes. To address such instabilities, we leverage optimization
analysis and propose an LLM structural pruning method, called MoreauPruner,
with provable robustness against weight perturbations. In MoreauPruner, the
model weight importance is estimated based on the neural network's Moreau
envelope, which can be flexibly combined with $\ell_1$-norm regularization
techniques to induce the sparsity required in the pruning task. We extensively
evaluate the MoreauPruner algorithm on several well-known LLMs, including
LLaMA-7B, LLaMA-13B, LLaMA3-8B, and Vicuna-7B. Our numerical results suggest
the robustness of MoreauPruner against weight perturbations, and indicate the
MoreauPruner's successful accuracy-based scores in comparison to several
existing pruning methods. We have released the code in
https://github.com/ShiningSord/MoreauPruner.

在大型语言模型中，考虑到模型权重的扰动效应，我们通过优化分析和 Moreau 包络来提出了一种名为 MoreauPruner 的结构剪枝方法，能够稳定地对模型进行剪枝，并成功地与其他几种剪枝方法进行了比较。