Vision Transformers (ViTs) have emerged as the fundamental architecture for
most computer vision fields, but the considerable memory and computation costs
hinders their application on resource-limited devices. As one of the most
powerful compression methods, binarization reduces the computation of the
neural network by quantizing the weights and activation values as $\pm$1.
Although existing binarization methods have demonstrated excellent performance
on Convolutional Neural Networks (CNNs), the full binarization of ViTs is still
under-studied and suffering a significant performance drop. In this paper, we
first argue empirically that the severe performance degradation is mainly
caused by the weight oscillation in the binarization training and the
information distortion in the activation of ViTs. Based on these analyses, we
propose $\textbf{BinaryViT}$, an accurate full binarization scheme for ViTs,
which pushes the quantization of ViTs to the limit. Specifically, we propose a
novel gradient regularization scheme (GRS) for driving a bimodal distribution
of the weights to reduce oscillation in binarization training. Moreover, we
design an activation shift module (ASM) to adaptively tune the activation
distribution to reduce the information distortion caused by binarization.
Extensive experiments on ImageNet dataset show that our BinaryViT consistently
surpasses the strong baseline by 2.05% and improve the accuracy of fully
binarized ViTs to a usable level. Furthermore, our method achieves impressive
savings of 16.2$\times$ and 17.7$\times$ in model size and OPs compared to the
full-precision DeiT-S. The codes and models will be released on github.

本论文提出了一种名为 BinaryViT 的方法，通过引入梯度正则化方案和激活偏移模块解决了 Vision Transformers 全量二值化所面临的性能降低和信息失真问题，使得其在计算和存储资源受限的设备上也能有更好的运行表现。