Action recognition in videos poses a challenge due to its high computational
cost, especially for Joint Space-Time video transformers (Joint VT). Despite
their effectiveness, the excessive number of tokens in such architectures
significantly limits their efficiency. In this paper, we propose HaltingVT, an
efficient video transformer adaptively removing redundant video patch tokens,
which is primarily composed of a Joint VT and a Glimpser module. Specifically,
HaltingVT applies data-adaptive token reduction at each layer, resulting in a
significant reduction in the overall computational cost. Besides, the Glimpser
module quickly removes redundant tokens in shallow transformer layers, which
may even be misleading for video recognition tasks based on our observations.
To further encourage HaltingVT to focus on the key motion-related information
in videos, we design an effective Motion Loss during training. HaltingVT
acquires video analysis capabilities and token halting compression strategies
simultaneously in a unified training process, without requiring additional
training procedures or sub-networks. On the Mini-Kinetics dataset, we achieved
75.0% top-1 ACC with 24.2 GFLOPs, as well as 67.2% top-1 ACC with an extremely
low 9.9 GFLOPs. The code is available at
this https URL

在这篇论文中，我们提出了 HaltingVT，一种高效的视频转换器，通过自适应地删除多余的视频补丁令牌，显著降低了计算成本。此外，我们设计了一种有效的运动损失函数，在训练期间进一步鼓励 HaltingVT 专注于视频中的关键动作相关信息。通过在 Mini-Kinetics 数据集上的实验，我们在极低的 9.9 GFLOPs 下达到了 67.2% 的 top-1 准确率。