Large language models (LLMs) have shown remarkable capabilities in various
tasks. However their huge model size and the consequent demand for
computational and memory resources also pose challenges to model deployment.
Currently, 4-bit post-training quantization (PTQ) has achieved some success in
LLMs, reducing the memory footprint by approximately 75% compared to FP16
models, albeit with some accuracy loss. In this paper, we propose SmoothQuant+,
an accurate and efficient 4-bit weight-only PTQ that requires no additional
training, which enables lossless in accuracy for LLMs for the first time. Based
on the fact that the loss of weight quantization is amplified by the activation
outliers, SmoothQuant+ smoothes the activation outliers by channel before
quantization, while adjusting the corresponding weights for mathematical
equivalence, and then performs group-wise 4-bit weight quantization for linear
layers. We have integrated SmoothQuant+ into the vLLM framework, an advanced
high-throughput inference engine specially developed for LLMs, and equipped it
with an efficient W4A16 CUDA kernels, so that vLLM can seamlessly support
SmoothQuant+ 4-bit weight quantization. Our results show that, with
SmoothQuant+, the Code Llama-34B model can be quantized and deployed on a A100
40GB GPU, achieving lossless accuracy and a throughput increase of 1.9 to 4.0
times compared to the FP16 model deployed on two A100 40GB GPUs. Moreover, the
latency per token is only 68% of the FP16 model deployed on two A100 40GB GPUs.
This is the state-of-the-art 4-bit weight quantization for LLMs as we know.

提出了 SmoothQuant + 方法，它是一种准确而高效的 4 位权重量化方法，能够无损地减小大语言模型的内存开销，并且在精确度上没有损失。通过 SmoothQuant+，Code Llama-34B 模型能够在一张 A100 40GB GPU 上实现无损的准确度，并且相较于在两张 A100 40GB GPUs 上部署的 FP16 模型，能够提高 1.9 至 4.0 倍的吞吐量，每个 token 的延迟仅为 FP16 模型的 68%。这是已知的大语言模型 4 位权重量化的最先进方法。