Video transformers have recently emerged as a competitive alternative to 3D
CNNs for video understanding. However, due to their large number of parameters
and reduced inductive biases, these models require supervised pretraining on
large-scale image datasets to achieve top performance. In this paper, we
empirically demonstrate that self-supervised pretraining of video transformers
on video-only datasets can lead to action recognition results that are on par
or better than those obtained with supervised pretraining on large-scale image
datasets, even massive ones such as ImageNet-21K. Since transformer-based
models are effective at capturing dependencies over extended temporal spans, we
propose a simple learning procedure that forces the model to match a long-term
view to a short-term view of the same video. Our approach, named Long-Short
Temporal Contrastive Learning (LSTCL), enables video transformers to learn an
effective clip-level representation by predicting temporal context captured
from a longer temporal extent. To demonstrate the generality of our findings,
we implement and validate our approach under three different self-supervised
contrastive learning frameworks (MoCo v3, BYOL, SimSiam) using two distinct
video-transformer architectures, including an improved variant of the Swin
Transformer augmented with space-time attention. We conduct a thorough ablation
study and show that LSTCL achieves competitive performance on multiple video
benchmarks and represents a convincing alternative to supervised image-based
pretraining.

本文提出了一种叫做 “长短时对比学习（LSTCL）” 的学习程序，可以让视频变压器模型在预测来自更长的时间范围内捕获的时序上下文的同时，学习一个有效的剪辑级表示，并在多项视频基准测试上取得了有竞争力的表现，成为有监督基于图像的预训练的有力替代方案。