Recent transformer language models achieve outstanding results in many
natural language processing (NLP) tasks. However, their enormous size often
makes them impractical on memory-constrained devices, requiring practitioners
to compress them to smaller networks. In this paper, we explore offline
compression methods, meaning computationally-cheap approaches that do not
require further fine-tuning of the compressed model. We challenge the classical
matrix factorization methods by proposing a novel, better-performing
autoencoder-based framework. We perform a comprehensive ablation study of our
approach, examining its different aspects over a diverse set of evaluation
settings. Moreover, we show that enabling collaboration between modules across
layers by compressing certain modules together positively impacts the final
model performance. Experiments on various NLP tasks demonstrate that our
approach significantly outperforms commonly used factorization-based offline
compression methods.

本篇文章旨在探讨离线压缩方法，并提出了一种基于自编码器的新框架，旨在压缩 transformer 语言模型。实验证明，启用模块间的协作并使用所提出的方法能够显著提高模型性能。