Video understanding relies on perceiving the global content and modeling its
internal connections (e.g., causality, movement, and spatio-temporal
correspondence). To learn these interactions, we apply a mask-then-predict
pre-training task on discretized video tokens generated via VQ-VAE. Unlike
language, where the text tokens are more independent, neighboring video tokens
typically have strong correlations (e.g., consecutive video frames usually look
very similar), and hence uniformly masking individual tokens will make the task
too trivial to learn useful representations. To deal with this issue, we
propose a block-wise masking strategy where we mask neighboring video tokens in
both spatial and temporal domains. We also add an augmentation-free contrastive
learning method to further capture the global content by predicting whether the
video clips are sampled from the same video. We pre-train our model on
uncurated videos and show that our pre-trained model can reach state-of-the-art
results on several video understanding datasets (e.g., SSV2, Diving48). Lastly,
we provide detailed analyses on model scalability and pre-training method
design. Code is released at this https URL

该研究基于预测任务以及块状掩码策略，提出一种输入处理策略及无数据扩充方法，以达到在 SSV2、Diving48 等视频理解数据集上实现最先进效果的目的，并对模型伸缩性和预训练方法进行了详细分析。

VIMPAC: 基于掩码预测和对比学习的视频预训练

VIMPAC: Video Pre-Training via Masked Token Prediction and Contrastive  Learning

We propose a method to learn image representations from uncurated videos. We
combine a supervised loss from off-the-shelf object detectors and
self-supervised losses which naturally arise from the video-shot-frame-object
hierarchy present in each video. We report competitive results on 19 transfer
learning tasks of the Visual Task Adaptation Benchmark (VTAB), and on 8
out-of-distribution-generalization tasks, and discuss the benefits and
shortcomings of the proposed approach. In particular, it improves over the
baseline on all 18/19 few-shot learning tasks and 8/8 out-of-distribution
generalization tasks. Finally, we perform several ablation studies and analyze
the impact of the pretrained object detector on the performance across this
suite of tasks.

该篇论文提出了一种从未加工过的视频中学习图像表示的方法，该方法将来自现成物体检测器的监督损失和自我监督损失相结合，取得了在 19 个迁移学习任务中有竞争力的结果，其中包括 18/19 的少样本学习任务和 8/8 的数据集泛化任务。