Contrastive learning has shown promising potential in self-supervised
spatio-temporal representation learning. Most works naively sample different
clips to construct positive and negative pairs. However, we observe that this
formulation inclines the model towards the background scene bias. The
underlying reasons are twofold. First, the scene difference is usually more
noticeable and easier to discriminate than the motion difference. Second, the
clips sampled from the same video often share similar backgrounds but have
distinct motions. Simply regarding them as positive pairs will draw the model
to the static background rather than the motion pattern. To tackle this
challenge, this paper presents a novel dual contrastive formulation.
Concretely, we decouple the input RGB video sequence into two complementary
modes, static scene and dynamic motion. Then, the original RGB features are
pulled closer to the static features and the aligned dynamic features,
respectively. In this way, the static scene and the dynamic motion are
simultaneously encoded into the compact RGB representation. We further conduct
the feature space decoupling via activation maps to distill static- and
dynamic-related features. We term our method as \textbf{D}ual
\textbf{C}ontrastive \textbf{L}earning for spatio-temporal
\textbf{R}epresentation (DCLR). Extensive experiments demonstrate that DCLR
learns effective spatio-temporal representations and obtains state-of-the-art
or comparable performance on UCF-101, HMDB-51, and Diving-48 datasets.

该论文提出了一种新的双重对比学习框架来解决背景场景偏差问题并有效地学习时空表示，实验证明在 UCF-101，HMDB-51 和 Diving-48 数据集上具有最先进的性能。