Token compression expedites the training and inference of Vision Transformers (ViTs) by reducing the number of the redundant tokens, e.g., pruning inattentive tokens or merging similar tokens. However, when applied to downstream tasks, these approaches suffer from significant performance drop when the compression degrees are mismatched between training and inference stages, which limits the application of token compression on off-the-shelf trained models. In this paper, we propose a model arithmetic framework to decouple the compression degrees between the two stages. In advance, we additionally perform a fast parameter-efficient self-distillation stage on the pre-trained models to obtain a small plugin, called Token Compensator (ToCom), which describes the gap between models across different compression degrees. During inference, ToCom can be directly inserted into any downstream off-the-shelf models with any mismatched training and inference compression degrees to acquire universal performance improvements without further training. Experiments on over 20 downstream tasks demonstrate the effectiveness of our framework. On CIFAR100, fine-grained visual classification, and VTAB-1k, ToCom can yield up to a maximum improvement of 2.3%, 1.5%, and 2.0% in the average performance of DeiT-B, respectively. Code: https://github.com/JieShibo/ToCom

本研究针对视觉变压器（ViTs）在训练和推理阶段压缩程度不匹配导致性能大幅下滑的问题，提出了一种模型算术框架，能够解耦这两个阶段的压缩程度。通过引入一种小插件——令牌补偿器（ToCom），可在推理时直接应用于任何下游模型，显著提升了模型的鲁棒性和性能。实验结果显示，在多个下游任务上，ToCom的应用可实现平均性能提高，验证了该方法的有效性。

令牌补偿器：在不重新调优的情况下改变视觉变压器的推理成本