There has been significant interest in "extreme" compression of large
language models (LLMs), i.e., to 1-2 bits per parameter, which allows such
models to be executed efficiently on resource-constrained devices. Existing
work focused on improved one-shot quantization techniques and weight
representations; yet, purely post-training approaches are reaching diminishing
returns in terms of the accuracy-vs-bit-width trade-off. State-of-the-art
quantization methods such as QuIP# and AQLM include fine-tuning (part of) the
compressed parameters over a limited amount of calibration data; however, such
fine-tuning techniques over compressed weights often make exclusive use of
straight-through estimators (STE), whose performance is not well-understood in
this setting. In this work, we question the use of STE for extreme LLM
compression, showing that it can be sub-optimal, and perform a systematic study
of quantization-aware fine-tuning strategies for LLMs. We propose PV-Tuning - a
representation-agnostic framework that generalizes and improves upon existing
fine-tuning strategies, and provides convergence guarantees in restricted
cases. On the practical side, when used for 1-2 bit vector quantization,
PV-Tuning outperforms prior techniques for highly-performant models such as
Llama and Mistral. Using PV-Tuning, we achieve the first Pareto-optimal
quantization for Llama 2 family models at 2 bits per parameter.

该研究提出了 PV-Tuning，一个对于极端压缩语言模型的量化感知微调策略的表示无关框架，通过在实践中应用于 1-2 位矢量量化，PV-Tuning 在高性能模型如 Llama 和 Mistral 中优于先前的技术，实现了 Llama 2 系列模型每个参数 2 位的帕累托最优量化。

PV-Tuning：用于极高 LLM 压缩的超越直通估计的方法

PV-Tuning: Beyond Straight-Through Estimation for Extreme LLM  Compression

Extreme compression, particularly ultra-low bit precision (binary/ternary)
quantization, has been proposed to fit large NLP models on resource-constraint
devices. However, to preserve the accuracy for such aggressive compression
schemes, cutting-edge methods usually introduce complicated compression
pipelines, e.g., multi-stage expensive knowledge distillation with extensive
hyperparameter tuning. Also, they oftentimes focus less on smaller transformer
models that have already been heavily compressed via knowledge distillation and
lack a systematic study to show the effectiveness of their methods. In this
paper, we perform a very comprehensive systematic study to measure the impact
of many key hyperparameters and training strategies from previous works. As a
result, we find out that previous baselines for ultra-low bit precision
quantization are significantly under-trained. Based on our study, we propose a
simple yet effective compression pipeline for extreme compression, named XTC.
XTC demonstrates that (1) we can skip the pre-training knowledge distillation
to obtain a 5-layer BERT while achieving better performance than previous
state-of-the-art methods, e.g., the 6-layer TinyBERT; (2) extreme quantization
plus layer reduction is able to reduce the model size by 50x, resulting in new
state-of-the-art results on GLUE tasks.

本文通过回顾相关学者先前的工作，系统地探讨了几个关键参数的影响。在此基础上，提出了一种名为 XTC 的简单而有效的压缩流程，它通过极端量化和层减少使模型尺寸降低了 50 倍，并在 GLUE 任务上实现了最新的最佳结果。

预训练 Transformer 的极致压缩简单高效实现

Extreme Compression for Pre-trained Transformers Made Simple and Efficient

In this paper, we present a novel adversarial lossy video compression model.
At extremely low bit-rates, standard video coding schemes suffer from
unpleasant reconstruction artifacts such as blocking, ringing etc. Existing
learned neural approaches to video compression have achieved reasonable success
on reducing the bit-rate for efficient transmission and reduce the impact of
artifacts to an extent. However, they still tend to produce blurred results
under extreme compression. In this paper, we present a deep adversarial learned
video compression model that minimizes an auxiliary adversarial distortion
objective. We find this adversarial objective to correlate better with human
perceptual quality judgement relative to traditional quality metrics such as
MS-SSIM and PSNR. Our experiments using a state-of-the-art learned video
compression system demonstrate a reduction of perceptual artifacts and
reconstruction of detail lost especially under extremely high compression.

本文提出了一种新颖的深度对抗学习视频压缩模型，该模型采用一个辅助的对抗性失真目标函数，我们将该目标函数与传统的质量指标如 MS-SSIM 和 PSNR 相比较，发现该对抗目标函数更符合人类感知质量判断标准，实验结果证明该模型能够在极高的压缩率下减少感知失真和重建丢失的细节。