As emerging hardware begins to support mixed bit-width arithmetic
computation, mixed-precision quantization is widely used to reduce the
complexity of neural networks. However, Vision Transformers (ViTs) require
complex self-attention computation to guarantee the learning of powerful
feature representations, which makes mixed-precision quantization of ViTs still
challenging. In this paper, we propose a novel patch-wise mixed-precision
quantization (PMQ) for efficient inference of ViTs. Specifically, we design a
lightweight global metric, which is faster than existing methods, to measure
the sensitivity of each component in ViTs to quantization errors. Moreover, we
also introduce a pareto frontier approach to automatically allocate the optimal
bit-precision according to the sensitivity. To further reduce the computational
complexity of self-attention in inference stage, we propose a patch-wise module
to reallocate bit-width of patches in each layer. Extensive experiments on the
ImageNet dataset shows that our method greatly reduces the search cost and
facilitates the application of mixed-precision quantization to ViTs.

本文提出了一种基于 patch 的混合精度量化方法，用于 ViTs 神经网络的高效推理，通过引入轻量级全局度量并采用帕累托前沿法自动分配最佳的位精度来减少搜索成本。实验证明这种方法大大减少了搜索成本，使混合精度量化应用于 ViTs 变得容易。