The pre-trained neural models have recently achieved impressive performances
in understanding multimodal content. However, it is still very challenging to
pre-train neural models for video and language understanding, especially for
Chinese video-language data, due to the following reasons. Firstly, existing
video-language pre-training algorithms mainly focus on the co-occurrence of
words and video frames, but ignore other valuable semantic and structure
information of video-language content, e.g., sequential order and
spatiotemporal relationships. Secondly, there exist conflicts between video
sentence alignment and other proxy tasks. Thirdly, there is a lack of
large-scale and high-quality Chinese video-language datasets (e.g., including
10 million unique videos), which are the fundamental success conditions for
pre-training techniques.
In this work, we propose a novel video-language understanding framework named
VICTOR, which stands for VIdeo-language understanding via Contrastive
mulTimOdal pRe-training. Besides general proxy tasks such as masked language
modeling, VICTOR constructs several novel proxy tasks under the contrastive
learning paradigm, making the model be more robust and able to capture more
complex multimodal semantic and structural relationships from different
perspectives. VICTOR is trained on a large-scale Chinese video-language
dataset, including over 10 million complete videos with corresponding
high-quality textual descriptions. We apply the pre-trained VICTOR model to a
series of downstream applications and demonstrate its superior performances,
comparing against the state-of-the-art pre-training methods such as VideoBERT
and UniVL. The codes and trained checkpoints will be publicly available to
nourish further developments of the research community.

本文提出了一种新型的视频 - 语言理解框架 VICTOR，通过对比学习范式构建了几个新型代理任务，使模型在不同角度能够更稳健并能够捕捉更多复杂的多模态语义和结构关系。通过在一个包括 10,000,000 个视频及其相应高质量文本描述的大规模中文视频语言数据集上训练 VICTOR，我们在一系列下游应用中展示了其优越的性能，并与 VideoBERT 和 UniVL 等先进的预训练方法进行了比较。