Self-supervised approaches for video have shown impressive results in video
understanding tasks. However, unlike early works that leverage temporal
self-supervision, current state-of-the-art methods primarily rely on tasks from
the image domain (e.g., contrastive learning) that do not explicitly promote
the learning of temporal features. We identify two factors that limit existing
temporal self-supervision: 1) tasks are too simple, resulting in saturated
training performance, and 2) we uncover shortcuts based on local appearance
statistics that hinder the learning of high-level features. To address these
issues, we propose 1) a more challenging reformulation of temporal
self-supervision as frame-level (rather than clip-level) recognition tasks and
2) an effective augmentation strategy to mitigate shortcuts. Our model extends
a representation of single video frames, pre-trained through contrastive
learning, with a transformer that we train through temporal self-supervision.
We demonstrate experimentally that our more challenging frame-level task
formulations and the removal of shortcuts drastically improve the quality of
features learned through temporal self-supervision. The generalization
capability of our self-supervised video method is evidenced by its
state-of-the-art performance in a wide range of high-level semantic tasks,
including video retrieval, action classification, and video attribute
recognition (such as object and scene identification), as well as low-level
temporal correspondence tasks like video object segmentation and pose tracking.
Additionally, we show that the video representations learned through our method
exhibit increased robustness to the input perturbations.

我们提出了更具挑战性的帧级自我监督任务和有效的增强策略，通过增加 Transformer 模型来训练从对比学习中预训练的单帧视频表示，大幅提升了通过时间自我监督学习到的特征的质量，并且在高水平语义任务和低水平时序任务上表现出卓越的性能。

不再取巧：实现时间自监督的潜力

No More Shortcuts: Realizing the Potential of Temporal Self-Supervision

Our work explores temporal self-supervision for GAN-based video generation
tasks. While adversarial training successfully yields generative models for a
variety of areas, temporal relationships in the generated data are much less
explored. Natural temporal changes are crucial for sequential generation tasks,
e.g. video super-resolution and unpaired video translation. For the former,
state-of-the-art methods often favor simpler norm losses such as $L^2$ over
adversarial training. However, their averaging nature easily leads to
temporally smooth results with an undesirable lack of spatial detail. For
unpaired video translation, existing approaches modify the generator networks
to form spatio-temporal cycle consistencies. In contrast, we focus on improving
learning objectives and propose a temporally self-supervised algorithm. For
both tasks, we show that temporal adversarial learning is key to achieving
temporally coherent solutions without sacrificing spatial detail. We also
propose a novel Ping-Pong loss to improve the long-term temporal consistency.
It effectively prevents recurrent networks from accumulating artifacts
temporally without depressing detailed features. Additionally, we propose a
first set of metrics to quantitatively evaluate the accuracy as well as the
perceptual quality of the temporal evolution. A series of user studies confirm
the rankings computed with these metrics. Code, data, models, and results are
provided at this https URL The project page
this https URL contains supplemental
materials.

本文提出了一种使用时间自我监督的算法，采用 GAN 生成视频的方式解决了时间关系难以探索的问题，同时提出了 Ping-Pong loss 用来改善长期时间一致性问题，并且介绍了一组评估模型的定量指标。