We consider the problem of accurate quantization for language models, where
both the weights and activations are uniformly quantized to 4 bits per
parameter, the lowest bitwidth format natively supported by GPU hardware. In
this context, the key challenge is activation quantization: it is known that
language models contain outlier channels whose values on average are orders of
magnitude higher than than other channels, which prevents accurate low-bitwidth
quantization with known techniques. We systematically study this phenomena and
find that these outlier channels emerge early in training, and that they occur
more frequently in layers with residual streams. We then propose a simple
strategy which regularizes a layer's inputs via quantization-aware training
(QAT) and its outputs via activation kurtosis regularization. We show that
regularizing both the inputs and outputs is crucial for preventing a model's
"migrating" the difficulty in input quantization to the weights, which makes
post-training quantization (PTQ) of weights more difficult. When combined with
weight PTQ, we show that our approach can obtain a W4A4 model that performs
competitively to the standard-precision W16A16 baseline.

准确的量化是语言模型的关键问题之一，我们研究了激活量化中的异常通道现象，并提出了一种调节输入和输出的策略，通过量化感知训练和激活峰度正则化来实现精确的 4 位参数量化。与权重后训练量化相结合，我们的方法可以获得与标准精度基准相竞争的 W4A4 模型。