Singular value decomposition (SVD) is one of the most popular compression
methods that approximate a target matrix with smaller matrices. However,
standard SVD treats the parameters within the matrix with equal importance,
which is a simple but unrealistic assumption. The parameters of a trained
neural network model may affect task performance unevenly, which suggests
non-equal importance among the parameters. Compared to SVD, the decomposition
method aware of parameter importance is the more practical choice in real
cases. Unlike standard SVD, weighted value decomposition is a non-convex
optimization problem that lacks a closed-form solution. We systematically
investigated multiple optimization strategies to tackle the problem and
examined our method by compressing Transformer-based language models. Further,
we designed a metric to predict when the SVD may introduce a significant
performance drop, for which our method can be a rescue strategy. The extensive
evaluations demonstrate that our method can perform better than current SOTA
methods in compressing Transformer-based language models.

我们提出了一种加权奇异值分解压缩 Transformer 的语言模型的方法，该方法考虑了神经网络参数的不平等重要性，并解决了没有封闭形式解决方案的非凸优化问题。实验结果表明，相较于传统的 SVD 方法，在压缩 Transformer 的语言模型时，我们的方法可以获得更好的效果。