The advancements in Large Language Models (LLMs) have been hindered by their
substantial sizes, which necessitate LLM compression methods for practical
deployment. Singular Value Decomposition (SVD) offers a promising solution for
LLM compression. However, state-of-the-art SVD-based LLM compression methods
have two key limitations: truncating smaller singular values may lead to higher
compression loss, and the lack of update on the remaining model parameters
after SVD truncation. In this work, we propose SVD-LLM, a new SVD-based LLM
compression method that addresses the limitations of existing methods. SVD-LLM
incorporates a truncation-aware data whitening strategy to ensure a direct
mapping between singular values and compression loss. Moreover, SVD-LLM adopts
a layer-wise closed-form model parameter update strategy to compensate for
accuracy degradation caused by SVD truncation. We evaluate SVD-LLM on a total
of 11 datasets and seven models from three different LLM families at four
different scales. Our results demonstrate the superiority of SVD-LLM over
state-of-the-arts, especially at high model compression ratios. The source code
is available at this https URL

提出了一种新的基于奇异值分解的大型语言模型压缩方法 SVD-LLM，它解决了现有方法的限制，并在高模型压缩比下展现了优越性能。