Vision Transformers (ViTs) have revolutionized the field of computer vision,
yet their deployments on resource-constrained devices remain challenging due to
high computational demands. To expedite pre-trained ViTs, token pruning and
token merging approaches have been developed, which aim at reducing the number
of tokens involved in the computation. However, these methods still have some
limitations, such as image information loss from pruned tokens and inefficiency
in the token-matching process. In this paper, we introduce a novel Graph-based
Token Propagation (GTP) method to resolve the challenge of balancing model
efficiency and information preservation for efficient ViTs. Inspired by graph
summarization algorithms, GTP meticulously propagates less significant tokens'
information to spatially and semantically connected tokens that are of greater
importance. Consequently, the remaining few tokens serve as a summarization of
the entire token graph, allowing the method to reduce computational complexity
while preserving essential information of eliminated tokens. Combined with an
innovative token selection strategy, GTP can efficiently identify image tokens
to be propagated. Extensive experiments have validated GTP's effectiveness,
demonstrating both efficiency and performance improvements. Specifically, GTP
decreases the computational complexity of both DeiT-S and DeiT-B by up to 26%
with only a minimal 0.3% accuracy drop on ImageNet-1K without finetuning, and
remarkably surpasses the state-of-the-art token merging method on various
backbones at an even faster inference speed. The source code is available at
this https URL

基于图的标记传播方法（GTP）通过将不重要的标记信息传播到与之相关性更大的标记上，从而在降低计算复杂度的同时保持了被剔除标记的重要信息，实现了高效的 Vision Transformers（ViTs）。