Network quantization significantly reduces model inference complexity and has
been widely used in real-world deployments. However, most existing quantization
methods have been developed mainly on Convolutional Neural Networks (CNNs), and
suffer severe degradation when applied to fully quantized vision transformers.
In this work, we demonstrate that many of these difficulties arise because of
serious inter-channel variation in LayerNorm inputs, and present, Power-of-Two
Factor (PTF), a systematic method to reduce the performance degradation and
inference complexity of fully quantized vision transformers. In addition,
observing an extreme non-uniform distribution in attention maps, we propose
Log-Int-Softmax (LIS) to sustain that and simplify inference by using 4-bit
quantization and the BitShift operator. Comprehensive experiments on various
transformer-based architectures and benchmarks show that our Fully Quantized
Vision Transformer (FQ-ViT) outperforms previous works while even using lower
bit-width on attention maps. For instance, we reach 84.89% top-1 accuracy with
ViT-L on ImageNet and 50.8 mAP with Cascade Mask R-CNN (Swin-S) on COCO. To our
knowledge, we are the first to achieve lossless accuracy degradation (~1%) on
fully quantized vision transformers. The code is available at
this https URL

本研究开发了基于 Power-of-Two Factor（PTF）和 Log-Int-Softmax（LIS）的技术来简化全量化视觉 Transformer 的推理复杂度，并应用于各种基于 Transformer 的体系结构和基准测试中，以达到更高的性能。