Recent large-scale video-language pre-trained models have shown appealing
performance on various downstream tasks. However, the pre-training process is
computationally expensive due to the requirement of millions of video-text
pairs and the redundant data structure of each video. To mitigate these
problems, we propose LiteVL, which adapts a pre-trained image-language model
BLIP into a video-text model directly on downstream tasks, without heavy
pre-training. To enhance the temporal modeling lacking in the image-language
model, we propose to add temporal attention modules in the image encoder of
BLIP with dynamic temporal scaling. Besides the model-wise adaptation, we also
propose a non-parametric pooling mechanism to adaptively reweight the
fine-grained video embedding conditioned on the text. Experimental results on
text-video retrieval and video question answering show that the proposed LiteVL
even outperforms previous video-language pre-trained models by a clear margin,
though without any video-language pre-training.

本文提出 LiteVL 模型，结合 BLIP 图像语言模型，通过使用动态时态缩放，给图像编码器添加时间注意力模块，并提出非参数池化机制，以自适应地重新加权文本条件下的细粒度视频嵌入，取得了良好的性能，即使没有进行任何视频 - 语言预训练。