As the size of large language models (LLMs) continues to grow, model
compression without sacrificing accuracy has become a crucial challenge for
deployment. While some quantization methods, such as GPTQ, have made progress
in achieving acceptable 4-bit weight-only quantization, attempts at lower bit
quantization often result in severe performance degradation. In this paper, we
introduce a technique called norm tweaking, which can be used as a plugin in
current PTQ methods to achieve high precision while being cost-efficient. Our
approach is inspired by the observation that rectifying the quantized
activation distribution to match its float counterpart can readily restore
accuracy for LLMs. To achieve this, we carefully design a tweaking strategy
that includes calibration data generation and channel-wise distance constraint
to update the weights of normalization layers for better generalization. We
conduct extensive experiments on various datasets using several open-sourced
LLMs. Our method demonstrates significant improvements in both weight-only
quantization and joint quantization of weights and activations, surpassing
existing PTQ methods. On GLM-130B and OPT-66B, our method even achieves the
same level of accuracy at 2-bit quantization as their float ones. Our simple
and effective approach makes it more practical for real-world applications.

在本文中，我们介绍了一种称为 norm tweaking 的技术，可以作为当前 PTQ 方法的插件使用，以实现高精度并且具有成本效益的模型压缩。通过校准数据生成和通道级距离约束来更新归一化层的权重，我们的方法在权重量化和权重与激活联合量化方面取得了显著的改进，在 2 位量化情况下甚至达到与浮点数模型相同的精度水平。我们的简单有效的方法使其在实际应用中更加实用。