In this paper, we propose self-supervised training for video transformers
using unlabeled video data. From a given video, we create local and global
spatiotemporal views with varying spatial sizes and frame rates. Our
self-supervised objective seeks to match the features of these different views
representing the same video, to be invariant to spatiotemporal variations in
actions. To the best of our knowledge, the proposed approach is the first to
alleviate the dependency on negative samples or dedicated memory banks in
Self-supervised Video Transformer (SVT). Further, owing to the flexibility of
Transformer models, SVT supports slow-fast video processing within a single
architecture using dynamically adjusted positional encoding and supports
long-term relationship modeling along spatiotemporal dimensions. Our approach
performs well on four action recognition benchmarks (Kinetics-400, UCF-101,
HMDB-51, and SSv2) and converges faster with small batch sizes. Code:
this https URL

本研究提出了一种基于未标记视频数据进行自监督训练的视频 Transformer 方法。通过使用不同的空间大小和帧速率创建局部和全局的时空视图，实现了视频内 actions 的时空不变性，并且该方法不再需要使用负样本或专用存储器。该方法在动作识别基准数据集（Kinetics-400，UCF-101，HMDB-51 和 SSv2）上表现出色，并能够在小批量的情况下更快地收敛。