Quantization can accelerate large language model (LLM) inference. Going
beyond INT8 quantization, the research community is actively exploring even
lower precision, such as INT4. Nonetheless, state-of-the-art INT4 quantization
techniques only accelerate low-batch, edge LLM inference, failing to deliver
performance gains in large-batch, cloud-based LLM serving. We uncover a
critical issue: existing INT4 quantization methods suffer from significant
runtime overhead (20-90%) when dequantizing either weights or partial sums on
GPUs. To address this challenge, we introduce QoQ, a W4A8KV4 quantization
algorithm with 4-bit weight, 8-bit activation, and 4-bit KV cache. QoQ stands
for quattuor-octo-quattuor, which represents 4-8-4 in Latin. QoQ is implemented
by the QServe inference library that achieves measured speedup. The key insight
driving QServe is that the efficiency of LLM serving on GPUs is critically
influenced by operations on low-throughput CUDA cores. Building upon this
insight, in QoQ algorithm, we introduce progressive quantization that can allow
low dequantization overhead in W4A8 GEMM. Additionally, we develop
SmoothAttention to effectively mitigate the accuracy degradation incurred by
4-bit KV quantization. In the QServe system, we perform compute-aware weight
reordering and take advantage of register-level parallelism to reduce
dequantization latency. We also make fused attention memory-bound, harnessing
the performance gain brought by KV4 quantization. As a result, QServe improves
the maximum achievable serving throughput of Llama-3-8B by 1.2x on A100, 1.4x
on L40S; and Qwen1.5-72B by 2.4x on A100, 3.5x on L40S, compared to
TensorRT-LLM. Remarkably, QServe on L40S GPU can achieve even higher throughput
than TensorRT-LLM on A100. Thus, QServe effectively reduces the dollar cost of
LLM serving by 3x. Code is available at this https URL

量化技术在提速大规模语言模型推理方面发挥作用，但现有的 INT4 量化方法在大批量云端应用中存在效率问题。本研究提出了一种名为 QoQ 的算法，该算法通过使用 4 位权重、8 位激活和 4 位 KV 缓存来解决该挑战，并通过 QServe 推理库实现。QServe 在 GPU 上通过低吞吐 CUDA 核心上的操作来提高 LLM 服务的效率，采用渐进量化来减小 W4A8 GEMM 的量化开销，并通过 SmoothAttention 减轻 4 位 KV 量化带来的精度下降，从而有效提高 LLM 的服务吞吐量，降低成本。

QServe: W4A8KV4 量化和系统协同设计用于高效 LLM 服务

QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM  Serving

This study examines 4-bit quantization methods like GPTQ in large language
models (LLMs), highlighting GPTQ's overfitting and limited enhancement in
Zero-Shot tasks. While prior works merely focusing on zero-shot measurement, we
extend task scope to more generative categories such as code generation and
abstractive summarization, in which we found that INT4 quantization can
significantly underperform. However, simply shifting to higher precision
formats like FP6 has been particularly challenging, thus overlooked, due to
poor performance caused by the lack of sophisticated integration and system
acceleration strategies on current AI hardware. Our results show that FP6, even
with a coarse-grain quantization scheme, performs robustly across various
algorithms and tasks, demonstrating its superiority in accuracy and
versatility. Notably, with the FP6 quantization, \codestar-15B model performs
comparably to its FP16 counterpart in code generation, and for smaller models
like the 406M it closely matches their baselines in summarization. Neither can
be achieved by INT4. To better accommodate various AI hardware and achieve the
best system performance, we propose a novel 4+2 design for FP6 to achieve
similar latency to the state-of-the-art INT4 fine-grain quantization. With our
design, FP6 can become a promising solution to the current 4-bit quantization
methods used in LLMs.

本研究审查了大型语言模型中的 4 位量化方法，重点关注 GPTQ 在零样本任务中的过拟合问题和限制。我们扩展了任务范围，涵盖了生成类别，如代码生成和抽象概括，发现 INT4 量化在其中显著性能不佳。然而，转向高精度格式如 FP6 却面临挑战，因为现有 AI 硬件上缺乏复杂的集成和系统加速策略导致性能较差，常被忽视。实验结果显示，FP6，即便采用粗粒度的量化方案，也能在各种算法和任务上表现出色，展示了它在准确性和多功能性方面的优势。值得注意的是，使用 FP6 量化，codestar-15B 模型在代码生成方面表现与其 FP16 对等，对于较小的模型如 406M，与摘要生成的基准模型非常接近，而 INT4 无法达到这种性能。为了更好适应各种 AI 硬件并实现最佳系统性能，我们提出了一种用于 FP6 的新颖 4+2 设计，以达到与最先进的 INT4 精细的量化相似的时延。通过我们的设计，FP6 可以成为当前 LLMs 中使用的 4 位量化方法的有希望的解决方案。