Recent advances in large language model (LLM) pretraining have led to
high-quality LLMs with impressive abilities. By compressing such LLMs via
quantization to 3-4 bits per parameter, they can fit into memory-limited
devices such as laptops and mobile phones, enabling personalized use. However,
quantization down to 3-4 bits per parameter usually leads to moderate-to-high
accuracy losses, especially for smaller models in the 1-10B parameter range,
which are well-suited for edge deployments. To address this accuracy issue, we
introduce the Sparse-Quantized Representation (SpQR), a new compressed format
and quantization technique which enables for the first time near-lossless
compression of LLMs across model scales, while reaching similar compression
levels to previous methods. SpQR works by identifying and isolating outlier
weights, which cause particularly-large quantization errors, and storing them
in higher precision, while compressing all other weights to 3-4 bits, and
achieves relative accuracy losses of less than 1% in perplexity for
highly-accurate LLaMA and Falcon LLMs. This makes it possible to run 33B
parameter LLM on a single 24 GB consumer GPU without any performance
degradation at 15% speedup thus making powerful LLMs available to consumer
without any downsides. SpQR comes with efficient algorithms for both encoding
weights into its format, as well as decoding them efficiently at runtime.
Specifically, we provide an efficient GPU inference algorithm for SpQR which
yields faster inference than 16-bit baselines at similar accuracy, while
enabling memory compression gains of more than 4x.

通过 Sparse-Quantized Representation（SpQR）压缩 LLMs，提出全新的压缩格式和量化技术，通过将所有其他权重压缩到 3-4 位并将异常权重以高精度存储以解决精度问题，既能保存模型的准确性，又能达到先前方法相似的压缩水平，该方法的运行速度要比 16 位基线快，并实现超过 4 倍的内存压缩收益。