The breakthrough performance of large language models (LLMs) comes with large
computational footprints and high deployment costs. In this paper, we progress
towards resolving this problem by proposing a new structured compression
approach for LLMs, called ZipLM, which provides state-of-the-art
compression-vs-accuracy results, while guaranteeing to match a set of
(achievable) target speedups on any given target hardware. Specifically, given
a task, a model, an inference environment, as well as a set of speedup targets,
ZipLM identifies and removes redundancies in the model through iterative
structured shrinking of the model's weight matrices. Importantly, ZipLM works
in both, the post-training/one-shot and the gradual compression setting, where
it produces a set of accurate models in a single run, making it
highly-efficient in practice. Our approach is based on new structured pruning
and knowledge distillation techniques, and consistently outperforms prior
structured compression methods in terms of accuracy-versus-speedup in
experiments on BERT- and GPT-family models. In particular, when compressing
GPT2 model, it outperforms DistilGPT2 while being 60% smaller and 30% faster.
Further, ZipLM matches performance of heavily optimized MobileBERT model,
obtained via extensive architecture search, by simply pruning the baseline
BERT-large architecture, and outperforms all prior BERT-base compression
techniques like CoFi, MiniLM and TinyBERT.

该研究提出了一种新的大规模语言模型结构压缩方法 ZipLM，通过迭代的结构缩小模型的权重矩阵，实现在保证一组可行的目标加速比的同时提供最先进的压缩精度结果，并且能够在单次运行中在后训练 / 一次性和渐进压缩设置下生成一组准确的模型，其基于新的结构剪枝和知识蒸馏技术，结果表明 ZipLM 优于先前的结构压缩方法，尤其在 GPT2 模型上表现最佳。