Large language models (LLMs) have shown excellent performance on various
tasks, but the astronomical model size raises the hardware barrier for serving
(memory size) and slows down token generation (memory bandwidth). In this
paper, we propose Activation-aware Weight Quantization (AWQ), a
hardware-friendly approach for LLM low-bit weight-only quantization. Our method
is based on the observation that weights are not equally important: protecting
only 1% of salient weights can greatly reduce quantization error. We then
propose to search for the optimal per-channel scaling that protects the salient
weights by observing the activation, not weights. AWQ does not rely on any
backpropagation or reconstruction, so it can well preserve LLMs' generalization
ability on different domains and modalities, without overfitting to the
calibration set; it also does not rely on any data layout reordering,
maintaining the hardware efficiency. AWQ outperforms existing work on various
language modeling, common sense QA, and domain-specific benchmarks. Thanks to
better generalization, it achieves excellent quantization performance for
instruction-tuned LMs and, for the first time, multi-modal LMs. We also
implement efficient tensor core kernels with reorder-free online dequantization
to accelerate AWQ, achieving a 1.45x speedup over GPTQ and is 1.85x faster than
the cuBLAS FP16 implementation. Our method provides a turn-key solution to
compress LLMs to 3/4 bits for efficient deployment.

提出了一种基于激活函数的、硬件友好型的低比特权重量化方法。该方法可以保护神经网络的关键权重并保持其泛化能力，在不影响硬件效率的前提下，有效降低了语言模型部署的成本。