Recently, the efficient deployment and acceleration of powerful vision transformers (ViTs) on resource-limited edge devices for providing multimedia services have become attractive tasks. Although early exiting is a feasible solution for accelerating inference, most works focus on convolutional neural networks (CNNs) and transformer models in natural language processing (NLP).Moreover, the direct application of early exiting methods to ViTs may result in substantial performance degradation. To tackle this challenge, we systematically investigate the efficacy of early exiting in ViTs and point out that the insufficient feature representations in shallow internal classifiers and the limited ability to capture target semantic information in deep internal classifiers restrict the performance of these methods. We then propose an early exiting framework for general ViTs termed LGViT, which incorporates heterogeneous exiting heads, namely, local perception head and global aggregation head, to achieve an efficiency-accuracy trade-off. In particular, we develop a novel two-stage training scheme, including end-to-end training and self-distillation with the backbone frozen to generate early exiting ViTs, which facilitates the fusion of global and local information extracted by the two types of heads. We conduct extensive experiments using three popular ViT backbones on three vision datasets. Results demonstrate that our LGViT can achieve competitive performance with approximately 1.8 $\times$ speed-up.

我们提出了一种早期退出的通用ViTs框架LGViT，它通过引入异质退出头，包括局部感知头和全局聚合头，实现了效率和准确性的权衡，通过两阶段的训练方案，包括端到端训练和带有冻结骨干的自蒸馏，生成了早期退出的ViTs，进一步促进了由这两种类型的头提取的全局和局部信息的融合，实验证明我们的LGViT能够在大约1.8倍的速度提升的同时保持竞争力的性能。

LGViT：动态早期退出以加速视觉Transformer