Neural network quantization aims to accelerate and trim full-precision neural network models by using low bit approximations. Methods adopting the quantization aware training (QAT) paradigm have recently seen a rapid growth, but are often conceptually complicated. This paper proposes a novel and highly effective QAT method, quantized feature distillation (QFD). QFD first trains a quantized (or binarized) representation as the teacher, then quantize the network using knowledge distillation (KD). Quantitative results show that QFD is more flexible and effective (i.e., quantization friendly) than previous quantization methods. QFD surpasses existing methods by a noticeable margin on not only image classification but also object detection, albeit being much simpler. Furthermore, QFD quantizes ViT and Swin-Transformer on MS-COCO detection and segmentation, which verifies its potential in real world deployment. To the best of our knowledge, this is the first time that vision transformers have been quantized in object detection and image segmentation tasks.

神经网络量化是通过使用低比特近似来加速和剪裁全精度神经网络模型的过程。本文提出了一种新颖且高效的量化感知训练方法，即量化特征蒸馏（QFD），通过首先训练一个量化（或二值化）的教师表示，然后使用知识蒸馏（KD）来量化网络。定量结果表明，QFD比之前的量化方法更加灵活和有效（即量化友好），在图像分类和目标检测任务上明显优于现有方法，同时又更为简单。此外，QFD对MS-COCO detection和segmentation中的ViT和Swin-Transformer进行了量化验证，从而验证了其在实际部署中的潜力。据我们所知，这是第一次将视觉变换器量化应用于目标检测和图像分割任务。

网络量化的特征量化蒸馏