The quality of the image representations obtained from self-supervised
learning depends strongly on the type of data augmentations used in the
learning formulation. Recent papers have ported these methods from still images
to videos and found that leveraging both audio and video signals yields strong
gains; however, they did not find that spatial augmentations such as cropping,
which are very important for still images, work as well for videos. In this
paper, we improve these formulations in two ways unique to the spatio-temporal
aspect of videos. First, for space, we show that spatial augmentations such as
cropping do work well for videos too, but that previous implementations, due to
the high processing and memory cost, could not do this at a scale sufficient
for it to work well. To address this issue, we first introduce Feature Crop, a
method to simulate such augmentations much more efficiently directly in feature
space. Second, we show that as opposed to naive average pooling, the use of
transformer-based attention improves performance significantly, and is well
suited for processing feature crops. Combining both of our discoveries into a
new method, Space-Time Crop & Attend (STiCA) we achieve state-of-the-art
performance across multiple video-representation learning benchmarks. In
particular, we achieve new state-of-the-art accuracies of 67.0% on HMDB-51 and
93.1% on UCF-101 when pre-training on Kinetics-400.

本文提出了一种基于特征裁剪和注意力机制的视频自监督学习方法 STiCA，改进了当前视频自监督学习方法中对于空间维度数据增强的应用以及特征融合方式，取得了多项数据集上的最优结果。