Large language models (LLMs) have exhibited exciting progress in multiple
scenarios, while the huge computational demands hinder their deployments in
lots of real-world applications. As an effective means to reduce memory
footprint and inference cost, quantization also faces challenges in performance
degradation at low bit-widths. Understanding the impact of quantization on LLM
capabilities, especially the generalization ability, is crucial. However, the
community's main focus remains on the algorithms and models of quantization,
with insufficient attention given to whether the quantized models can retain
the strong generalization abilities of LLMs. In this work, we fill this gap by
providing a comprehensive benchmark suite for this research topic, including an
evaluation system, detailed analyses, and a general toolbox. Specifically,
based on the dominant pipeline in LLM quantization, we primarily explore the
impact of calibration data distribution on the generalization of quantized LLMs
and conduct the benchmark using more than 40 datasets within two main
scenarios. Based on this benchmark, we conduct extensive experiments with two
well-known LLMs (English and Chinese) and four quantization algorithms to
investigate this topic in-depth, yielding several counter-intuitive and
valuable findings, e.g., models quantized using a calibration set with the same
distribution as the test data are not necessarily optimal. Besides, to
facilitate future research, we also release a modular-designed toolbox, which
decouples the overall pipeline into several separate components, e.g., base LLM
module, dataset module, quantizer module, etc. and allows subsequent
researchers to easily assemble their methods through a simple configuration.
Our benchmark suite is publicly available at
this https URL

使用全面的基准套件，我们探索了大规模语言模型在量化过程中的泛化能力以及校准数据分布对其影响，并且发布了一个模块化设计的工具包来帮助未来的研究。