Vision transformers (ViTs) have become the popular structures and
outperformed convolutional neural networks (CNNs) on various vision tasks.
However, such powerful transformers bring a huge computation burden, because of
the exhausting token-to-token comparison. The previous works focus on dropping
insignificant tokens to reduce the computational cost of ViTs. But when the
dropping ratio increases, this hard manner will inevitably discard the vital
tokens, which limits its efficiency. To solve the issue, we propose a generic
self-slimmed learning approach for vanilla ViTs, namely SiT. Specifically, we
first design a novel Token Slimming Module (TSM), which can boost the inference
efficiency of ViTs by dynamic token aggregation. As a general method of token
hard dropping, our TSM softly integrates redundant tokens into fewer
informative ones. It can dynamically zoom visual attention without cutting off
discriminative token relations in the images, even with a high slimming ratio.
Furthermore, we introduce a concise Feature Recalibration Distillation (FRD)
framework, wherein we design a reverse version of TSM (RTSM) to recalibrate the
unstructured token in a flexible auto-encoder manner. Due to the similar
structure between teacher and student, our FRD can effectively leverage
structure knowledge for better convergence. Finally, we conduct extensive
experiments to evaluate our SiT. It demonstrates that our method can speed up
ViTs by 1.7x with negligible accuracy drop, and even speed up ViTs by 3.6x
while maintaining 97% of their performance. Surprisingly, by simply arming
LV-ViT with our SiT, we achieve new state-of-the-art performance on ImageNet.
Code is available at this https URL

该研究提出了一种基于 Token Slimming Module 和 Feature Recalibration Distillation 的自我瘦身学习方法（SiT），能够大幅减少 Vision Transformers 的计算负担，同时不会对其分类性能产生太大影响，并在 ImageNet 数据集上实现了新的最佳性能。