Post-training quantization (PTQ) reduces the memory footprint of LLMs by
quantizing their weights to low-precision. In this work, we introduce QuIP#, a
weight-only PTQ method that achieves state-of-the-art results in extreme
compression regimes ($\le$ 4 bits per weight) using three novel techniques.
First, QuIP# improves the incoherence processing from QuIP by using the
randomized Hadamard transform, which is faster and has better theoretical
properties. Second, QuIP# uses vector quantization techniques to take advantage
of the ball-shaped sub-Gaussian distribution that incoherent weights possess:
specifically, we introduce a set of hardware-efficient codebooks based on the
highly symmetric $E_8$ lattice, which achieves the optimal 8-dimension unit
ball packing. Third, QuIP# uses fine-tuning to improve fidelity to the original
model. Our experiments show that QuIP# outperforms existing PTQ methods,
enables new behaviors in PTQ scaling, and supports fast inference.

在这项研究中，我们引入了权重量化的新方法（QuIP#），通过三个创新技术，在极端压缩模式下（每个权重≤4 位）实现了最先进的结果。QuIP# 通过使用随机哈达玛变换改进 QuIP 的不相干处理，并利用具有不相干权重的球形亚高斯分布来使用向量量化技术。此外，QuIP# 使用微调来提高对原始模型的保真度，实验结果显示其超过了现有的 PTQ 方法，支持快速推理。