Large Language Models (LLMs) pose significant hardware challenges related to
memory requirements and computational ability. There are two mainstream
quantization schemes for LLMs: coarse-grained ($\textit{e.g.,}$ channel-wise)
quantization and fine-grained ($\textit{e.g.,}$ group-wise) quantization.
Fine-grained quantization has smaller quantization loss, consequently achieving
superior performance. However, when applied to weight-activation quantization,
it disrupts continuous integer matrix multiplication, leading to inefficient
inference. In this paper, we introduce Dual Grained Quantization (DGQ), a novel
A8W4 quantization for LLM that maintains superior performance while ensuring
fast inference speed. DSQ dequantizes the fine-grained INT4 weight into
coarse-grained INT8 representation and preform matrix multiplication using INT8
kernels. Besides, we develop a two-phase grid search algorithm to simplify the
determination of fine-grained and coarse-grained quantization scales. We also
devise a percentile clipping schema for smoothing the activation outliers
without the need for complex optimization techniques. Experimental results
demonstrate that DGQ consistently outperforms prior methods across various LLM
architectures and a wide range of tasks. Remarkably, by our implemented
efficient CUTLASS kernel, we achieve $\textbf{1.12}$ $\times$ memory reduction
and $\textbf{3.24}$ $\times$ speed gains comparing A16W4 implementation. These
advancements enable efficient deployment of A8W4 LLMs for real-world
applications.

该论文介绍了一种称为 Dual Grained Quantization (DGQ) 的新型量化技术，通过将细粒度的 INT4 权重解量化为粗粒度的 INT8 表示，并使用 INT8 内核进行矩阵乘法，来保持卓越性能同时确保快速推理速度。实验结果表明，DGQ 在各种 LLM 架构和广泛的任务中始终优于之前的方法，通过高效的 CUTLASS 内核，实现 1.12 倍的内存减少和 3.24 倍的速度增益，从而实现了 A8W4 LLM 在实际应用中的高效部署。