We present CLIP2Video network to transfer the image-language pre-training
model to video-text retrieval in an end-to-end manner. Leading approaches in
the domain of video-and-language learning try to distill the spatio-temporal
video features and multi-modal interaction between videos and languages from a
large-scale video-text dataset. Different from them, we leverage pretrained
image-language model, simplify it as a two-stage framework with co-learning of
image-text and enhancing temporal relations between video frames and video-text
respectively, make it able to train on comparatively small datasets.
Specifically, based on the spatial semantics captured by Contrastive
Language-Image Pretraining (CLIP) model, our model involves a Temporal
Difference Block to capture motions at fine temporal video frames, and a
Temporal Alignment Block to re-align the tokens of video clips and phrases and
enhance the multi-modal correlation. We conduct thorough ablation studies, and
achieve state-of-the-art performance on major text-to-video and video-to-text
retrieval benchmarks, including new records of retrieval accuracy on MSR-VTT,
MSVD and VATEX.

CLIP2Video 网络通过将图像语言预训练模型转移到视频文本检索，采用端到端方式，区别于领先的视频和语言学习方法的多模态交互，我们利用预训练的图像语言模型，进一步简化为两个具体阶段的框架，使其能够在相对较少的数据集上进行训练，并通过 Temporal Difference Block 和 Temporal Alignment Block 来提升多模态相关性，我们在 MSR-VTT，MSVD 和 VATEX 等主要文本到视频和视频到文本检索基准上实现了最新的检索准确性记录。