Quantization-aware training (QAT) and Knowledge Distillation (KD) are
combined to achieve competitive performance in creating low-bit deep learning
models. However, existing works applying KD to QAT require tedious
hyper-parameter tuning to balance the weights of different loss terms, assume
the availability of labeled training data, and require complex, computationally
intensive training procedures for good performance. To address these
limitations, this paper proposes a novel Self-Supervised Quantization-Aware
Knowledge Distillation (SQAKD) framework. SQAKD first unifies the forward and
backward dynamics of various quantization functions, making it flexible for
incorporating various QAT works. Then it formulates QAT as a co-optimization
problem that simultaneously minimizes the KL-Loss between the full-precision
and low-bit models for KD and the discretization error for quantization,
without supervision from labels. A comprehensive evaluation shows that SQAKD
substantially outperforms the state-of-the-art QAT and KD works for a variety
of model architectures. Our code is at: this https URL

该论文提出了一种新颖的自监督量化感知知识蒸馏 (SQAKD) 框架，通过将量化感知训练与知识蒸馏相结合，同时最小化全精度模型与低比特模型之间的 KL-Loss 和量化的离散化误差，从而有效克服了现有方法需要繁琐的超参数调节、标签数据需求和复杂的训练过程的限制，并在各种模型结构上明显优于现有的量化感知训练和知识蒸馏方法。