We propose a novel self-supervised method, referred to as Video Cloze Procedure (VCP), to learn rich spatial-temporal representations. VCP first generates "blanks" by withholding video clips and then creates "options" by applying spatio-temporal operations on the withheld clips. Finally, it fills the blanks with "options" and learns representations by predicting the categories of operations applied on the clips. VCP can act as either a proxy task or a target task in self-supervised learning. As a proxy task, it converts rich self-supervised representations into video clip operations (options), which enhances the flexibility and reduces the complexity of representation learning. As a target task, it can assess learned representation models in a uniform and interpretable manner. With VCP, we train spatial-temporal representation models (3D-CNNs) and apply such models on action recognition and video retrieval tasks. Experiments on commonly used benchmarks show that the trained models outperform the state-of-the-art self-supervised models with significant margins.

该论文提出了一种自我监督学习的方法——视频Cloze过程（VCP），用于学习丰富的时空表示。通过将其作为代理任务或目标任务，VCP可以将自我监督表示转换为视频剪辑操作（选项），从而提高了表示学习的灵活性并降低了复杂性，并可以以一种统一且可解释的方式评估已学习的表示模型，实验结果表明，与最先进的自监督模型相比，所训练的模型具有显著优势。

自监督时空学习的视频空缺程序