Large Language Models (LLMs) are reshaping the research landscape in
artificial intelligence, particularly as model parameters scale up
significantly, unlocking remarkable capabilities across various domains.
Nevertheless, the scalability of model parameters faces constraints due to
limitations in GPU memory and computational speed. To address these
constraints, various weight compression methods have emerged, such as Pruning
and Quantization. Given the low-rank nature of weight matrices in language
models, the reduction of weights through matrix decomposition undoubtedly holds
significant potential and promise. In this paper, drawing upon the intrinsic
structure of LLMs, we propose a novel approach termed Data-free Joint Rank-k
Approximation for compressing the parameter matrices. Significantly, our method
is characterized by without necessitating additional involvement of any corpus,
while simultaneously preserving orthogonality in conjunction with pruning and
quantization methods. We achieve a model pruning of 80% parameters while
retaining 93.43% of the original performance without any calibration data.
Additionally, we explore the fundamental properties of the weight matrix of
LLMs undergone Rank-k Approximation and conduct comprehensive experiments to
elucidate our hypothesis.

基于大型语言模型的权重分解以及压缩方法，通过新的无需语料库参与、保持正交性的数据无关联 Rank-k 近似方法，成功压缩了 80% 的参数并保留了原始性能的 93.43%。同时，对经过 Rank-k 近似的权重矩阵进行了深入研究以验证假设。

大型语言模型的零数据压缩和降噪

Data-freeWeight Compress and Denoise for Large Language Models

We prove that for any real-valued matrix $X \in \R^{m \times n}$, and
positive integers $r \ge k$, there is a subset of $r$ columns of $X$ such that
projecting $X$ onto their span gives a $\sqrt{\frac{r+1}{r-k+1}}$-approximation
to best rank-$k$ approximation of $X$ in Frobenius norm. We show that the
trade-off we achieve between the number of columns and the approximation ratio
is optimal up to lower order terms. Furthermore, there is a deterministic
algorithm to find such a subset of columns that runs in $O(r n m^{\omega} \log
m)$ arithmetic operations where $\omega$ is the exponent of matrix
multiplication. We also give a faster randomized algorithm that runs in $O(r n
m^2)$ arithmetic operations.

证明对于任意实值矩阵 X，在正整数 r≥k 的情况下，存在 X 的 r 个列的子集，将 X 投影到其中一个列的线性组合中，得出的结果将是 Frobenius 范数下的 X 的最佳秩 - k 逼近的一个近似值等于  sqrt ((r + 1)/(r-k +1))。并提出一个确定性算法可在  O (r n m^ω log m) 时间内找到这样的列的子集，其中 ω 是矩阵乘法的指数。同样，我们提供了一种更快的随机算法，可在 O (r n m^2) 时间内进行运算。