Vision Transformers (ViTs) have demonstrated outstanding performance in computer vision tasks, yet their high computational complexity prevents their deployment in computing resource-constrained environments. Various token pruning techniques have been introduced to alleviate the high computational burden of ViTs by dynamically dropping image tokens. However, some undesirable pruning at early stages may result in permanent loss of image information in subsequent layers, consequently hindering model performance. To address this problem, we propose IdleViT, a dynamic token-idle-based method that achieves an excellent trade-off between performance and efficiency. Specifically, in each layer, IdleViT selects a subset of the image tokens to participate in computations while keeping the rest of the tokens idle and directly passing them to this layer's output. By allowing the idle tokens to be re-selected in the following layers, IdleViT mitigates the negative impact of improper pruning in the early stages. Furthermore, inspired by the normalized graph cut, we devise a token cut loss on the attention map as regularization to improve IdleViT's token selection ability. Our method is simple yet effective and can be extended to pyramid ViTs since no token is completely dropped. Extensive experimental results on various ViT architectures have shown that IdleViT can diminish the complexity of pretrained ViTs by up to 33\% with no more than 0.2\% accuracy decrease on ImageNet, after finetuning for only 30 epochs. Notably, when the keep ratio is 0.5, IdleViT outperforms the state-of-the-art EViT on DeiT-S by 0.5\% higher accuracy and even faster inference speed. The source code is available in the supplementary material.

为了解决Vision Transformers在计算资源受限环境中的高计算复杂度问题，该研究提出了IdleViT方法，通过动态删除图像令牌来改善计算负担，实现了性能和效率之间的卓越折中。在各个层中，IdleViT选择参与计算的图像令牌子集，并将其余的令牌保持空闲并直接传递给该层的输出，通过在后续层重新选择空闲令牌，IdleViT减轻了早期错误剪枝对模型性能的负面影响。该方法简单而有效，可扩展到金字塔ViTs，因为没有完全删除的令牌。大量的实验结果表明，IdleViT可以将预训练ViTs的复杂性降低高达33％，在只进行30个周期的微调之后在ImageNet上的准确率降低不超过0.2％。值得注意的是，在保留比例为0.5时，IdleViT相比于最先进的EViT在DeiT-S上的准确率更高，且推理速度更快。

高效的视觉Transformer模型：动态空闲令牌策略