Vision transformers have achieved remarkable success in computer vision tasks
by using multi-head self-attention modules to capture long-range dependencies
within images. However, the high inference computation cost poses a new
challenge. Several methods have been proposed to address this problem, mainly
by slimming patches. In the inference stage, these methods classify patches
into two classes, one to keep and the other to discard in multiple layers. This
approach results in additional computation at every layer where patches are
discarded, which hinders inference acceleration.
In this study, we tackle the patch slimming problem from a different
perspective by proposing a life regression module that determines the lifespan
of each image patch in one go. During inference, the patch is discarded once
the current layer index exceeds its life. Our proposed method avoids additional
computation and parameters in multiple layers to enhance inference speed while
maintaining competitive performance. Additionally, our approach requires fewer
training epochs than other patch slimming methods.

本研究提出了一个基于生命周期回归模块的图像分割方法，它可以在减少计算量及参数的情况下，提高推理速度，在保持竞争性能的同时需要较少的训练历程。

基于生命回归的视觉变压器局部细化

Life Regression based Patch Slimming for Vision Transformers

This paper studies the efficiency problem for visual transformers by
excavating redundant calculation in given networks. The recent transformer
architecture has demonstrated its effectiveness for achieving excellent
performance on a series of computer vision tasks. However, similar to that of
convolutional neural networks, the huge computational cost of vision
transformers is still a severe issue. Considering that the attention mechanism
aggregates different patches layer-by-layer, we present a novel patch slimming
approach that discards useless patches in a top-down paradigm. We first
identify the effective patches in the last layer and then use them to guide the
patch selection process of previous layers. For each layer, the impact of a
patch on the final output feature is approximated and patches with less impact
will be removed. Experimental results on benchmark datasets demonstrate that
the proposed method can significantly reduce the computational costs of vision
transformers without affecting their performances. For example, over 45% FLOPs
of the ViT-Ti model can be reduced with only 0.2% top-1 accuracy drop on the
ImageNet dataset.

该论文通过挖掘网络中的冗余计算研究视觉变换器的效率问题，并提出了一种新颖的修剪方法来减少计算成本，该方法称为修剪补丁法，可以移除无用的补丁，从而显著降低模型的计算成本，而不会影响模型的性能。