This paper presents a novel pre-trained language models (PLM) compression
approach based on the matrix product operator (short as MPO) from quantum
many-body physics. It can decompose an original matrix into central tensors
(containing the core information) and auxiliary tensors (with only a small
proportion of parameters). With the decomposed MPO structure, we propose a
novel fine-tuning strategy by only updating the parameters from the auxiliary
tensors, and design an optimization algorithm for MPO-based approximation over
stacked network architectures. Our approach can be applied to the original or
the compressed PLMs in a general way, which derives a lighter network and
significantly reduces the parameters to be fine-tuned. Extensive experiments
have demonstrated the effectiveness of the proposed approach in model
compression, especially the reduction in finetuning parameters (91% reduction
on average).

提出了一种基于量子多体物理中的矩阵积算符（简称 MPO）的新型预训练语言模型压缩方法，其可以将原始的矩阵分解成中心张量和辅助张量，通过仅更新辅助张量的参数进行微调并为 MPO 结构设计优化算法，可应用于原始或压缩的语言模型以实现更轻量级的网络和显著减少微调参数。