Post-training quantization (\ptq) had been recently shown as a compromising method to reduce the memory consumption and/or compute cost for large language models. However, a comprehensive study about the effect of different quantization schemes, different model families, different \ptq methods, different quantization bit precision, etc, is still missing. In this work, we provide an extensive study on those components over tens of thousands of zero-shot experiments. Our results show that (1) Fine-grained quantization and \ptq methods (instead of naive round-to-nearest quantization) are necessary to achieve good accuracy and (2) Higher bits (e.g., 5 bits) with coarse-grained quantization is more powerful than lower bits (e.g., 4 bits) with very fine-grained quantization (whose effective bits is similar to 5-bits). We also present recommendations about how to utilize quantization for \llms with different sizes, and leave suggestions of future opportunities and system work that are not resolved in this work.

通过数万次的零-shot实验，我们对后期训练量化(PTQ)的各种组成成分和效应进行了全面的研究，发现细粒度量化和PTQ方法是获得良好准确性所必需的，并且粗粒度量化的高比特位（例如，5比特）比非常细粒度量化的低比特位（例如，4比特）更强大。

大型语言模型的后训练量化综合研究