The increasing size of generative Pre-trained Language Models (PLMs) has
greatly increased the demand for model compression. Despite various methods to
compress BERT or its variants, there are few attempts to compress generative
PLMs, and the underlying difficulty remains unclear. In this paper, we compress
generative PLMs by quantization. We find that previous quantization methods
fail on generative tasks due to the \textit{homogeneous word embeddings} caused
by reduced capacity, and \textit{varied distribution of weights}.
Correspondingly, we propose a token-level contrastive distillation to learn
distinguishable word embeddings, and a module-wise dynamic scaling to make
quantizers adaptive to different modules. Empirical results on various tasks
show that our proposed method outperforms the state-of-the-art compression
methods on generative PLMs by a clear margin. With comparable performance with
the full-precision models, we achieve 14.4x and 13.4x compression rates on
GPT-2 and BART, respectively.

本文介绍了一种基于量化的压缩生成式语言模型的方法，该方法使用了基于 token 的对比 distillation 技术学习可区分的单词嵌入，并提出了一个基于模块的动态缩放技术，适应不同模块的量化器，实验结果表明该方法在各种任务上性能优于基准方法，并且实现了 14.4x 和 13.4x 的压缩率，与全精度模型具有可比性。