Despite the success of transformers on various computer vision tasks, they
suffer from excessive memory and computational cost. Some works present dynamic
vision transformers to accelerate inference by pruning redundant tokens. A key
to improving token pruning is using well-trained models as initialization for
faster convergence and better performance. However, current base models usually
adopt full image training, i.e., using full images as inputs and keeping the
whole feature maps through the forward process, which causes inconsistencies
with dynamic models that gradually reduce tokens, including calculation
pattern, information amount and token selection strategy inconsistencies.
Inspired by MAE which performs masking and reconstruction self-supervised task,
we devise masked fine-tuning to bridge the gaps between pre-trained base models
used for initialization and token pruning based dynamic vision transformers, by
masking image patches and predicting the image class label based on left
unmasked patches. Extensive experiments on ImageNet demonstrate that base
models via masked fine-tuning gain strong occlusion robustness and ability
against information loss. With this better initialization, Dynamic ViT achieves
higher accuracies, especially under large token pruning ratios (e.g., 81.9% vs.
81.3%, and 62.3% vs. 58.9% for DeiT based Dynamic ViT/0.8 and Dynamic ViT/0.3).
Moreover, we apply our method into different token pruning based dynamic vision
transformers, different pre-trained models and randomly initialized models to
demonstrate the generalization ability.

采用蒙版微调为动态视觉转换器的静态预训练基础模型提供更好的初始化，以提高准确性并增强其对遮挡的鲁棒性和对信息丢失的抵抗能力。