Large language models (LLMs) have garnered significant attention for their remarkable capabilities across various domains, whose vast parameter scales present challenges for practical deployment. Structured pruning is an effective method to balance model performance with efficiency, but performance restoration under computational resource constraints is a principal challenge in pruning LLMs. Therefore, we present a low-cost and fast structured pruning method for LLMs named SlimGPT based on the Optimal Brain Surgeon framework. We propose Batched Greedy Pruning for rapid and near-optimal pruning, which enhances the accuracy of head-wise pruning error estimation through grouped Cholesky decomposition and improves the pruning efficiency of FFN via Dynamic Group Size, thereby achieving approximate local optimal pruning results within one hour. Besides, we explore the limitations of layer-wise pruning from the perspective of error accumulation and propose Incremental Pruning Ratio, a non-uniform pruning strategy to reduce performance degradation. Experimental results on the LLaMA benchmark show that SlimGPT outperforms other methods and achieves state-of-the-art results.

本研究针对大型语言模型（LLMs）在实际应用中因参数规模庞大而面临的结构剪枝性能恢复难题，提出了一种低成本、高效的分层结构剪枝方法SlimGPT。该方法通过批量贪婪剪枝和增量剪枝比率，实现了快速近似最优的剪枝效果，实验结果表明，SlimGPT在LLaMA基准测试中表现优于其他方法，达到了当前最先进的水平。

SlimGPT：大型语言模型的分层结构剪枝