There has been significant interest in "extreme" compression of large
language models (LLMs), i.e., to 1-2 bits per parameter, which allows such
models to be executed efficiently on resource-constrained devices. Existing
work focused on improved one-shot quantization techniques and weight
representations; yet, purely post-training approaches are reaching diminishing
returns in terms of the accuracy-vs-bit-width trade-off. State-of-the-art
quantization methods such as QuIP# and AQLM include fine-tuning (part of) the
compressed parameters over a limited amount of calibration data; however, such
fine-tuning techniques over compressed weights often make exclusive use of
straight-through estimators (STE), whose performance is not well-understood in
this setting. In this work, we question the use of STE for extreme LLM
compression, showing that it can be sub-optimal, and perform a systematic study
of quantization-aware fine-tuning strategies for LLMs. We propose PV-Tuning - a
representation-agnostic framework that generalizes and improves upon existing
fine-tuning strategies, and provides convergence guarantees in restricted
cases. On the practical side, when used for 1-2 bit vector quantization,
PV-Tuning outperforms prior techniques for highly-performant models such as
Llama and Mistral. Using PV-Tuning, we achieve the first Pareto-optimal
quantization for Llama 2 family models at 2 bits per parameter.

该研究提出了 PV-Tuning，一个对于极端压缩语言模型的量化感知微调策略的表示无关框架，通过在实践中应用于 1-2 位矢量量化，PV-Tuning 在高性能模型如 Llama 和 Mistral 中优于先前的技术，实现了 Llama 2 系列模型每个参数 2 位的帕累托最优量化。