Large language models (LLMs) show excellent performance but are compute- and
memory-intensive. Quantization can reduce memory and accelerate inference.
However, for LLMs beyond 100 billion parameters, existing methods cannot
maintain accuracy or do not run efficiently on hardware. We propose
SmoothQuant, a training-free, accuracy-preserving, and general-purpose
post-training quantization (PTQ) solution to enable 8-bit weight, 8-bit
activation (W8A8) quantization for LLMs. Based on the fact that weights are
easy to quantize while activations are not, SmoothQuant smooths the activation
outliers by offline migrating the quantization difficulty from activations to
weights with a mathematically equivalent transformation. SmoothQuant enables an
INT8 quantization of both weights and activations for all the matrix
multiplications in LLMs, including OPT-175B, BLOOM-176B, GLM-130B, and MT-NLG
530B. SmoothQuant has better hardware efficiency than existing techniques. We
demonstrate up to 1.56x speedup and 2x memory reduction for LLMs with
negligible loss in accuracy. We integrate SmoothQuant into FasterTransformer, a
state-of-the-art LLM serving framework, and achieve faster inference speed with
half the number of GPUs compared to FP16, enabling the serving of a 530B LLM
within a single node. Our work offers a turn-key solution that reduces hardware
costs and democratizes LLMs. Code is available at
this https URL.

SmoothQuant 是一个训练免费的精度保持、通用的后训练量化解决方案，用于大型语言模型（LLMs），通过平滑激活异常值并在权重和激活之间进行数学上等效的变换以迁移量化难度，可以实现 LLMs 的 8 位权重和激活（W8A8）量化，同时提高硬件效率，以较小的精度损失实现高达 2 倍的内存减少和 1.56 倍的加速，是一个可降低硬件成本、民主化 LLMs 的一站式解决方案。