Large language models (LLMs) have demonstrated outstanding performance in
various tasks, such as text summarization, text question-answering, and etc.
While their performance is impressive, the computational footprint due to their
vast number of parameters can be prohibitive. Existing solutions such as
SparseGPT and Wanda attempt to alleviate this issue through weight pruning.
However, their layer-wise approach results in significant perturbation to the
model's output and requires meticulous hyperparameter tuning, such as the
pruning rate, which can adversely affect overall model performance. To address
this, this paper introduces a novel LLM pruning technique dubbed blockwise
parameter-efficient sparsity allocation (BESA) by applying a blockwise
reconstruction loss. In contrast to the typical layer-wise pruning techniques,
BESA is characterized by two distinctive attributes: i) it targets the overall
pruning error with respect to individual transformer blocks, and ii) it
allocates layer-specific sparsity in a differentiable manner, both of which
ensure reduced performance degradation after pruning. Our experiments show that
BESA achieves state-of-the-art performance, efficiently pruning LLMs like
LLaMA1, and LLaMA2 with 7B to 70B parameters on a single A100 GPU in just five
hours. Code is available at
\href{this https URL}{here}.

本论文提出了一种名为 BESA 的新颖大型语言模型修剪技术，通过施加分块重构损失来减少整体修剪误差并以可微分方式分配层特定的稀疏性，从而确保修剪后的模型性能降低程度减少，实验结果表明 BESA 在仅五小时内能够高效地修剪带有 70B 参数的 LLMs，达到了最先进的性能水平。