In recent years, large language models (LLMs) have driven advances in natural
language processing. Still, their growing scale has increased the computational
burden, necessitating a balance between efficiency and performance. Low-rank
compression, a promising technique, reduces non-essential parameters by
decomposing weight matrices into products of two low-rank matrices. Yet, its
application in LLMs has not been extensively studied. The key to low-rank
compression lies in low-rank factorization and low-rank dimensions allocation.
To address the challenges of low-rank compression in LLMs, we conduct empirical
research on the low-rank characteristics of large models. We propose a low-rank
compression method suitable for LLMs. This approach involves precise estimation
of feature distributions through pooled covariance matrices and a Bayesian
optimization strategy for allocating low-rank dimensions. Experiments on the
LLaMA-2 models demonstrate that our method outperforms existing strong
structured pruning and low-rank compression techniques in maintaining model
performance at the same compression ratio.

近年来，大型语言模型（LLMs）在自然语言处理方面取得了重要进展，但其规模的增大增加了计算负担，需要在效率和性能之间取得平衡。为了解决 LLMs 中低秩压缩的挑战，我们对大模型的低秩特性进行了实证研究，并提出了一种适用于 LLMs 的低秩压缩方法，该方法通过汇总协方差矩阵的特征分布精确估计和贝叶斯优化策略来分配低秩维度。我们在 LLaMA-2 模型上的实验表明，我们的方法在相同的压缩比下比现有的强结构剪枝和低秩压缩技术表现更佳，能够保持模型的性能。