The large pre-trained vision transformers (ViTs) have demonstrated remarkable
performance on various visual tasks, but suffer from expensive computational
and memory cost problems when deployed on resource-constrained devices. Among
the powerful compression approaches, quantization extremely reduces the
computation and memory consumption by low-bit parameters and bit-wise
operations. However, low-bit ViTs remain largely unexplored and usually suffer
from a significant performance drop compared with the real-valued counterparts.
In this work, through extensive empirical analysis, we first identify the
bottleneck for severe performance drop comes from the information distortion of
the low-bit quantized self-attention map. We then develop an information
rectification module (IRM) and a distribution guided distillation (DGD) scheme
for fully quantized vision transformers (Q-ViT) to effectively eliminate such
distortion, leading to a fully quantized ViTs. We evaluate our methods on
popular DeiT and Swin backbones. Extensive experimental results show that our
method achieves a much better performance than the prior arts. For example, our
Q-ViT can theoretically accelerates the ViT-S by 6.14x and achieves about 80.9%
Top-1 accuracy, even surpassing the full-precision counterpart by 1.0% on
ImageNet dataset. Our codes and models are attached on
this https URL

本文介绍了一种基于信息矫正模块和分布引导蒸馏方案的量化视觉变换器压缩方法，可以在 ImageNet 数据集上实现高达 80.9％的准确性，并且比全精度对应物超过 1.0％的 Top-1 精度。