Generative Large Language Models (LLMs) have demonstrated remarkable results
for a wide range of tasks. However, deploying these models for inference has
been a significant challenge due to their unprecedented resource requirements.
This has forced existing deployment frameworks to use multi-GPU inference
pipelines, which are often complex and costly, or to use smaller and less
performant models. In this work, we demonstrate that the main bottleneck for
generative inference with LLMs is memory bandwidth, rather than compute,
specifically for single batch inference. While quantization has emerged as a
promising solution by representing model weights with reduced precision,
previous efforts have often resulted in notable performance degradation. To
address this, we introduce SqueezeLLM, a post-training quantization framework
that not only enables lossless compression to ultra-low precisions of up to
3-bit, but also achieves higher quantization performance under the same memory
constraint. Our framework incorporates two novel ideas: (i) sensitivity-based
non-uniform quantization, which searches for the optimal bit precision
assignment based on second-order information; and (ii) the Dense-and-Sparse
decomposition that stores outliers and sensitive weight values in an efficient
sparse format. When applied to the LLaMA models, our 3-bit quantization
significantly reduces the perplexity gap from the FP16 baseline by up to 2.1x
as compared to the state-of-the-art methods with the same memory requirement.
Furthermore, when deployed on an A6000 GPU, our quantized models achieve up to
2.3x speedup compared to the baseline. Our code is open-sourced and available
online.

通过引入 SqueezeLLM 后训练的量化框架，该框架不仅实现了高达 3 位的无损压缩，还在相同的内存约束下实现了更高的量化性能，可以将羊毛出在羊身上，仿佛神器一般。