Vision transformers (ViTs) quantization offers a promising prospect to
facilitate deploying large pre-trained networks on resource-limited devices.
Fully-binarized ViTs (Bi-ViT) that pushes the quantization of ViTs to its limit
remain largely unexplored and a very challenging task yet, due to their
unacceptable performance. Through extensive empirical analyses, we identify the
severe drop in ViT binarization is caused by attention distortion in
self-attention, which technically stems from the gradient vanishing and ranking
disorder. To address these issues, we first introduce a learnable scaling
factor to reactivate the vanished gradients and illustrate its effectiveness
through theoretical and experimental analyses. We then propose a ranking-aware
distillation method to rectify the disordered ranking in a teacher-student
framework. Bi-ViT achieves significant improvements over popular DeiT and Swin
backbones in terms of Top-1 accuracy and FLOPs. For example, with DeiT-Tiny and
Swin-Tiny, our method significantly outperforms baselines by 22.1% and 21.4%
respectively, while 61.5x and 56.1x theoretical acceleration in terms of FLOPs
compared with real-valued counterparts on ImageNet.

介绍一种专门用于大规模预训练模型在硬件受限设备上快速部署的量化方法，并提出一种基于可训练缩放因子和排名感知蒸馏的 Binarized ViT 模型。该方法在 ImageNet 等数据集上都取得了较高的 Top-1 准确率，并在 FLOPs 方面获得了显著的理论加速。