Contrastive learning has shown great potential in video representation learning. However, existing approaches fail to sufficiently exploit short-term motion dynamics, which are crucial to various down-stream video understanding tasks. In this paper, we propose Motion Sensitive Contrastive Learning (MSCL) that injects the motion information captured by optical flows into RGB frames to strengthen feature learning. To achieve this, in addition to clip-level global contrastive learning, we develop Local Motion Contrastive Learning (LMCL) with frame-level contrastive objectives across the two modalities. Moreover, we introduce Flow Rotation Augmentation (FRA) to generate extra motion-shuffled negative samples and Motion Differential Sampling (MDS) to accurately screen training samples. Extensive experiments on standard benchmarks validate the effectiveness of the proposed method. With the commonly-used 3D ResNet-18 as the backbone, we achieve the top-1 accuracies of 91.5\% on UCF101 and 50.3\% on Something-Something v2 for video classification, and a 65.6\% Top-1 Recall on UCF101 for video retrieval, notably improving the state-of-the-art.

本文提出了一种Motion Sensitive Contrastive Learning (MSCL)方法，将光流获取的运动信息注入到RGB帧中，强化特征学习，通过局部运动对比学习（LMCL），以及流旋转增强（FRA）和运动差分采样（MDS）等方法对其进行扩展，对标准基准数据集进行了广泛的实验，提高了UCF101和Something-Something v2的视频分类性能，并显著提高了UCF101的视频检索性能。

基于运动敏感的对比学习方法用于自监督视频表征