While vision transformers (ViTs) have continuously achieved new milestones in
the field of computer vision, their sophisticated network architectures with
high computation and memory costs have impeded their deployment on
resource-limited edge devices. In this paper, we propose a hardware-efficient
image-adaptive token pruning framework called HeatViT for efficient yet
accurate ViT acceleration on embedded FPGAs. By analyzing the inherent
computational patterns in ViTs, we first design an effective attention-based
multi-head token selector, which can be progressively inserted before
transformer blocks to dynamically identify and consolidate the non-informative
tokens from input images. Moreover, we implement the token selector on hardware
by adding miniature control logic to heavily reuse existing hardware components
built for the backbone ViT. To improve the hardware efficiency, we further
employ 8-bit fixed-point quantization, and propose polynomial approximations
with regularization effect on quantization error for the frequently used
nonlinear functions in ViTs. Finally, we propose a latency-aware multi-stage
training strategy to determine the transformer blocks for inserting token
selectors and optimize the desired (average) pruning rates for inserted token
selectors, in order to improve both the model accuracy and inference latency on
hardware. Compared to existing ViT pruning studies, under the similar
computation cost, HeatViT can achieve 0.7%$\sim$8.9% higher accuracy; while
under the similar model accuracy, HeatViT can achieve more than
28.4%$\sim$65.3% computation reduction, for various widely used ViTs, including
DeiT-T, DeiT-S, DeiT-B, LV-ViT-S, and LV-ViT-M, on the ImageNet dataset.
Compared to the baseline hardware accelerator, our implementations of HeatViT
on the Xilinx ZCU102 FPGA achieve 3.46$\times$$\sim$4.89$\times$ speedup.

本文提出了一种硬件高效的图像自适应标记修剪框架 HeatViT，以在嵌入式 FPGA 上实现高效而准确的 ViT 加速，通过显著重复利用现有硬件组件来实现标记选择器，使用固定点量化和多阶段训练策略来优化插入标记选择器的变压器块，使模型在硬件上不仅提高了准确性和推理延迟，而且还可在与现有计算成本相似的情况下获得更高的准确性或在与相似模型准确性相似的情况下实现更高的计算量减少。