Contrastive learning has shown great potential in video representation
learning. However, existing approaches fail to sufficiently exploit short-term
motion dynamics, which are crucial to various down-stream video understanding
tasks. In this paper, we propose Motion Sensitive Contrastive Learning (MSCL)
that injects the motion information captured by optical flows into RGB frames
to strengthen feature learning. To achieve this, in addition to clip-level
global contrastive learning, we develop Local Motion Contrastive Learning
(LMCL) with frame-level contrastive objectives across the two modalities.
Moreover, we introduce Flow Rotation Augmentation (FRA) to generate extra
motion-shuffled negative samples and Motion Differential Sampling (MDS) to
accurately screen training samples. Extensive experiments on standard
benchmarks validate the effectiveness of the proposed method. With the
commonly-used 3D ResNet-18 as the backbone, we achieve the top-1 accuracies of
91.5\% on UCF101 and 50.3\% on Something-Something v2 for video classification,
and a 65.6\% Top-1 Recall on UCF101 for video retrieval, notably improving the
state-of-the-art.

本文提出了一种 Motion Sensitive Contrastive Learning (MSCL) 方法，将光流获取的运动信息注入到 RGB 帧中，强化特征学习，通过局部运动对比学习（LMCL），以及流旋转增强（FRA）和运动差分采样（MDS）等方法对其进行扩展，对标准基准数据集进行了广泛的实验，提高了 UCF101 和 Something-Something v2 的视频分类性能，并显著提高了 UCF101 的视频检索性能。