The breakthrough performance of large language models (LLMs) comes with large
computational footprints and high deployment costs. In this paper, we progress
towards resolving this problem by proposing a new structured compression
approach for LLMs, called ZipLM, which provides state-of-the-art
compression-vs-accuracy results, while guaranteeing to match a set of
(achievable) target speedups on any given target hardware. Specifically, given
a task, a model, an inference environment, as well as a set of speedup targets,
ZipLM identifies and removes redundancies in the model through iterative
structured shrinking of the model's weight matrices. Importantly, ZipLM works
in both, the post-training/one-shot and the gradual compression setting, where
it produces a set of accurate models in a single run, making it
highly-efficient in practice. Our approach is based on new structured pruning
and knowledge distillation techniques, and consistently outperforms prior
structured compression methods in terms of accuracy-versus-speedup in
experiments on BERT- and GPT-family models. In particular, when compressing
GPT2 model, it outperforms DistilGPT2 while being 60% smaller and 30% faster.
Further, ZipLM matches performance of heavily optimized MobileBERT model,
obtained via extensive architecture search, by simply pruning the baseline
BERT-large architecture, and outperforms all prior BERT-base compression
techniques like CoFi, MiniLM and TinyBERT.

该研究提出了一种新的大规模语言模型结构压缩方法 ZipLM，通过迭代的结构缩小模型的权重矩阵，实现在保证一组可行的目标加速比的同时提供最先进的压缩精度结果，并且能够在单次运行中在后训练 / 一次性和渐进压缩设置下生成一组准确的模型，其基于新的结构剪枝和知识蒸馏技术，结果表明 ZipLM 优于先前的结构压缩方法，尤其在 GPT2 模型上表现最佳。

ZipLM：面向硬件感知的语言模型结构化剪枝

ZipLM: Hardware-Aware Structured Pruning of Language Models

Recently, significant accuracy improvement has been achieved for acoustic
recognition systems by increasing the model size of Long Short-Term Memory
(LSTM) networks. Unfortunately, the ever-increasing size of LSTM model leads to
inefficient designs on FPGAs due to the limited on-chip resources. The previous
work proposes to use a pruning based compression technique to reduce the model
size and thus speedups the inference on FPGAs. However, the random nature of
the pruning technique transforms the dense matrices of the model to highly
unstructured sparse ones, which leads to unbalanced computation and irregular
memory accesses and thus hurts the overall performance and energy efficiency.
In contrast, we propose to use a structured compression technique which could
not only reduce the LSTM model size but also eliminate the irregularities of
computation and memory accesses. This approach employs block-circulant instead
of sparse matrices to compress weight matrices and reduces the storage
requirement from $\mathcal{O}(k^2)$ to $\mathcal{O}(k)$. Fast Fourier Transform
algorithm is utilized to further accelerate the inference by reducing the
computational complexity from $\mathcal{O}(k^2)$ to
$\mathcal{O}(k\text{log}k)$. The datapath and activation functions are
quantized as 16-bit to improve the resource utilization. More importantly, we
propose a comprehensive framework called C-LSTM to automatically optimize and
implement a wide range of LSTM variants on FPGAs. According to the experimental
results, C-LSTM achieves up to 18.8X and 33.5X gains for performance and energy
efficiency compared with the state-of-the-art LSTM implementation under the
same experimental setup, and the accuracy degradation is very small.

本文提出了一种基于块循环矩阵的压缩方案，用于改善长短时记忆网络的 FPGA 实现效率，并将该方法应用于一个名为 C-LSTM 的综合框架，其精度损失很小。