Large Language Models (LLMs) with a billion or more parameters are prime
targets for network pruning, which aims to reduce a portion of the network
weights without compromising performance. Prior approaches such as Weights
Magnitude, SparseGPT, and Wanda, either concentrated solely on weights or
integrated weights with activations for sparsity. However, they overlooked the
informative gradients derived from pretrained large language models. In this
paper, we present a novel sparsity-centric pruning method for pretrained LLMs,
termed Gradient-based Language Model Pruner (GBLM-Pruner). GBLM-Pruner
leverages the first-order term of the Taylor expansion, operating in a
training-free manner by harnessing properly normalized gradients from a few
calibration samples to determine the importance pruning score, and
substantially outperforms competitive counterparts like SparseGPT and Wanda in
multiple benchmarks. Intriguing, after incorporating gradients, the
unstructured pruning method tends to reveal some structural patterns
post-pruning, which mirrors the geometric interdependence inherent in the LLMs'
parameter structure. Additionally, GBLM-Pruner functions without any subsequent
retraining or weight updates to maintain its simplicity as other counterparts.
Extensive evaluations on LLaMA-1 and LLaMA-2 across various language benchmarks
and perplexity show that GBLM-Pruner surpasses magnitude pruning, Wanda
(weights+activations) and SparseGPT (weights+activations+weight update) by
significant margins. Our code and models are available at
this https URL

预训练的大型语言模型的梯度为基础的模型修剪器（GBLM-Pruner）通过利用卡尔曼几何中的几何相互关联性明显胜过其他竞争对手，并在各种语言评估中超过了幅度修剪、Wanda 和 SparseGPT。