Recently, Vision Transformer (ViT) has continuously established new
milestones in the computer vision field, while the high computation and memory
cost makes its propagation in industrial production difficult. Pruning, a
traditional model compression paradigm for hardware efficiency, has been widely
applied in various DNN structures. Nevertheless, it stays ambiguous on how to
perform exclusive pruning on the ViT structure. Considering three key points:
the structural characteristics, the internal data pattern of ViTs, and the
related edge device deployment, we leverage the input token sparsity and
propose a computation-aware soft pruning framework, which can be set up on
vanilla Transformers of both flatten and CNN-type structures, such as
Pooling-based ViT (PiT). More concretely, we design a dynamic attention-based
multi-head token selector, which is a lightweight module for adaptive
instance-wise token selection. We further introduce a soft pruning technique,
which integrates the less informative tokens generated by the selector module
into a package token that will participate in subsequent calculations rather
than being completely discarded. Our framework is bound to the trade-off
between accuracy and computation constraints of specific edge devices through
our proposed computation-aware training strategy. Experimental results show
that our framework significantly reduces the computation cost of ViTs while
maintaining comparable performance on image classification. Moreover, our
framework can guarantee the identified model to meet resource specifications of
mobile devices and FPGA, and even achieve the real-time execution of DeiT-T on
mobile platforms. For example, our method reduces the latency of DeiT-T to 26
ms (26%$\sim $41% superior to existing works) on the mobile device with
0.25%$\sim $4% higher top-1 accuracy on ImageNet.

通过利用输入令牌稀疏性并提出计算感知的软剪枝框架，可以大幅减少 Vision Transformer 计算成本，并满足移动设备和 FPGA 的资源规格要求，甚至在移动平台上实现 DeiT-T 的实时执行。