Video Transformers have become the prevalent solution for various video
downstream tasks with superior expressive power and flexibility. However, these
video transformers suffer from heavy computational costs induced by the massive
number of tokens across the entire video frames, which has been the major
barrier to training the model. Further, the patches irrelevant to the main
contents, e.g., backgrounds, degrade the generalization performance of models.
To tackle these issues, we propose training free token merging for lightweight
video Transformer (vid-TLDR) that aims to enhance the efficiency of video
Transformers by merging the background tokens without additional training. For
vid-TLDR, we introduce a novel approach to capture the salient regions in
videos only with the attention map. Further, we introduce the saliency-aware
token merging strategy by dropping the background tokens and sharpening the
object scores. Our experiments show that vid-TLDR significantly mitigates the
computational complexity of video Transformers while achieving competitive
performance compared to the base model without vid-TLDR. Code is available at
this https URL

提出了轻量级视频 Transformer（vid-TLDR）的训练免费的令牌合并策略，通过合并背景令牌并通过关注图捕捉显著区域来提高视频 Transformer 的效率。实验证明，vid-TLDR 显着减轻了视频 Transformer 的计算复杂度，并且在与基准模型相比的性能上取得了有竞争力的结果。