Self-supervised approaches for video have shown impressive results in video
understanding tasks. However, unlike early works that leverage temporal
self-supervision, current state-of-the-art methods primarily rely on tasks from
the image domain (e.g., contrastive learning) that do not explicitly promote
the learning of temporal features. We identify two factors that limit existing
temporal self-supervision: 1) tasks are too simple, resulting in saturated
training performance, and 2) we uncover shortcuts based on local appearance
statistics that hinder the learning of high-level features. To address these
issues, we propose 1) a more challenging reformulation of temporal
self-supervision as frame-level (rather than clip-level) recognition tasks and
2) an effective augmentation strategy to mitigate shortcuts. Our model extends
a representation of single video frames, pre-trained through contrastive
learning, with a transformer that we train through temporal self-supervision.
We demonstrate experimentally that our more challenging frame-level task
formulations and the removal of shortcuts drastically improve the quality of
features learned through temporal self-supervision. The generalization
capability of our self-supervised video method is evidenced by its
state-of-the-art performance in a wide range of high-level semantic tasks,
including video retrieval, action classification, and video attribute
recognition (such as object and scene identification), as well as low-level
temporal correspondence tasks like video object segmentation and pose tracking.
Additionally, we show that the video representations learned through our method
exhibit increased robustness to the input perturbations.

我们提出了更具挑战性的帧级自我监督任务和有效的增强策略，通过增加 Transformer 模型来训练从对比学习中预训练的单帧视频表示，大幅提升了通过时间自我监督学习到的特征的质量，并且在高水平语义任务和低水平时序任务上表现出卓越的性能。