Spatio-temporal representation learning is critical for video self-supervised
representation. Recent approaches mainly use contrastive learning and pretext
tasks. However, these approaches learn representation by discriminating sampled
instances via feature similarity in the latent space while ignoring the
intermediate state of the learned representations, which limits the overall
performance. In this work, taking into account the degree of similarity of
sampled instances as the intermediate state, we propose a novel pretext task -
spatio-temporal overlap rate (STOR) prediction. It stems from the observation
that humans are capable of discriminating the overlap rates of videos in space
and time. This task encourages the model to discriminate the STOR of two
generated samples to learn the representations. Moreover, we employ a joint
optimization combining pretext tasks with contrastive learning to further
enhance the spatio-temporal representation learning. We also study the mutual
influence of each component in the proposed scheme. Extensive experiments
demonstrate that our proposed STOR task can favor both contrastive learning and
pretext tasks. The joint optimization scheme can significantly improve the
spatio-temporal representation in video understanding. The code is available at
this https URL

本文提出了一种新的预训练任务方案，即基于空时重叠率 (STOR) 预测，旨在通过判断生成样本之间的 STOR 来学习视频自监督表示学习，并结合对比学习和预训练任务进行联合优化，从而进一步提高视频理解的空时表示学习。