Video transformers have achieved impressive results on major video
recognition benchmarks, which however suffer from high computational cost. In
this paper, we present STTS, a token selection framework that dynamically
selects a few informative tokens in both temporal and spatial dimensions
conditioned on input video samples. Specifically, we formulate token selection
as a ranking problem, which estimates the importance of each token through a
lightweight scorer network and only those with top scores will be used for
downstream evaluation. In the temporal dimension, we keep the frames that are
most relevant to the action categories, while in the spatial dimension, we
identify the most discriminative region in feature maps without affecting the
spatial context used in a hierarchical way in most video transformers. Since
the decision of token selection is non-differentiable, we employ a
perturbed-maximum based differentiable Top-K operator for end-to-end training.
We mainly conduct extensive experiments on Kinetics-400 with a recently
introduced video transformer backbone, MViT. Our framework achieves similar
results while requiring 20% less computation. We also demonstrate our approach
is generic for different transformer architectures and video datasets. Code is
available at this https URL

本文提出了 STTS 框架，这是一种动态选择少量视频关键信息的方法，可以在减小计算量的同时保持较高的识别准确率。

空间 - 时间令牌选择的有效视频 Transformer

Efficient Video Transformers with Spatial-Temporal Token Selection

Temporal action localization (TAL) is an important and challenging problem in
video understanding. However, most existing TAL benchmarks are built upon the
coarse granularity of action classes, which exhibits two major limitations in
this task. First, coarse-level actions can make the localization models overfit
in high-level context information, and ignore the atomic action details in the
video. Second, the coarse action classes often lead to the ambiguous
annotations of temporal boundaries, which are inappropriate for temporal action
localization. To tackle these problems, we develop a novel large-scale and
fine-grained video dataset, coined as FineAction, for temporal action
localization. In total, FineAction contains 103K temporal instances of 106
action categories, annotated in 17K untrimmed videos. Compared to the existing
TAL datasets, our FineAction takes distinct characteristics of fine action
classes with rich diversity, dense annotations of multiple instances, and
co-occurring actions of different classes, which introduces new opportunities
and challenges for temporal action localization. To benchmark FineAction, we
systematically investigate the performance of several popular temporal
localization methods on it, and deeply analyze the influence of fine-grained
instances in temporal action localization. As a minor contribution, we present
a simple baseline approach for handling the fine-grained action detection,
which achieves an mAP of 13.17% on our FineAction. We believe that FineAction
can advance research of temporal action localization and beyond.

开发了一个新的大规模、细粒度视频数据集 FineAction，共包含 106 个动作类别的 103k 个时间实例，在 17k 个未修剪的视频中进行了注释，用于对时间动作定位进行基准测试，并深入分析了细粒度实例对时间动作定位的影响。