The upscaling of Large Language Models (LLMs) has yielded impressive advances
in natural language processing, yet it also poses significant deployment
challenges. Weight quantization has emerged as a widely embraced solution to
reduce memory and computational demands. This paper introduces BitDistiller, a
framework that synergizes Quantization-Aware Training (QAT) with Knowledge
Distillation (KD) to boost the performance of LLMs at ultra-low precisions
(sub-4-bit). Specifically, BitDistiller first incorporates a tailored
asymmetric quantization and clipping technique to maximally preserve the
fidelity of quantized weights, and then proposes a novel Confidence-Aware
Kullback-Leibler Divergence (CAKLD) objective, which is employed in a
self-distillation manner to enable faster convergence and superior model
performance. Empirical evaluations demonstrate that BitDistiller significantly
surpasses existing methods in both 3-bit and 2-bit configurations on general
language understanding and complex reasoning benchmarks. Notably, BitDistiller
is shown to be more cost-effective, demanding fewer data and training
resources. The code is available at this https URL

BitDistiller 是一种通过 Quantization-Aware Training (QAT) 和 Knowledge Distillation (KD) 相结合的方法，可以提高大型语言模型在超低精度（低于 4 位）下的性能，包括了量化和剪裁技术以及一种新颖的 Confidence-Aware Kullback-Leibler Divergence (CAKLD) 目标函数，经实证评估，在 3 位和 2 位配置上明显超越现有方法，并且更加经济高效。