Transformer models have achieved remarkable results in various natural
language tasks, but they are often prohibitively large, requiring massive
memories and computational resources. To reduce the size and complexity of
these models, we propose LoSparse (Low-Rank and Sparse approximation), a novel
model compression technique that approximates a weight matrix by the sum of a
low-rank matrix and a sparse matrix. Our method combines the advantages of both
low-rank approximations and pruning, while avoiding their limitations. Low-rank
approximation compresses the coherent and expressive parts in neurons, while
pruning removes the incoherent and non-expressive parts in neurons. Pruning
enhances the diversity of low-rank approximations, and low-rank approximation
prevents pruning from losing too many expressive neurons. We evaluate our
method on natural language understanding, question answering, and natural
language generation tasks. We show that it significantly outperforms existing
compression methods.

本文提出了一种新颖的模型压缩技术 LoSparse，该技术通过低秩矩阵和稀疏矩阵的加和来近似一个权重矩阵，结合了低秩逼近和剪枝的优点，避免了它们的局限性，在自然语言理解、问答和生成等任务中，它显著优于现有的压缩方法。