Recent large-scale video-language pre-trained models have shown appealing
performance on various downstream tasks. However, the pre-training process is
computationally expensive due to the requirement of millions of video-text
pairs and the redundant data structure of each video. To mitigate these
problems, we propose LiteVL, which adapts a pre-trained image-language model
BLIP into a video-text model directly on downstream tasks, without heavy
pre-training. To enhance the temporal modeling lacking in the image-language
model, we propose to add temporal attention modules in the image encoder of
BLIP with dynamic temporal scaling. Besides the model-wise adaptation, we also
propose a non-parametric pooling mechanism to adaptively reweight the
fine-grained video embedding conditioned on the text. Experimental results on
text-video retrieval and video question answering show that the proposed LiteVL
even outperforms previous video-language pre-trained models by a clear margin,
though without any video-language pre-training.

本文提出 LiteVL 模型，结合 BLIP 图像语言模型，通过使用动态时态缩放，给图像编码器添加时间注意力模块，并提出非参数池化机制，以自适应地重新加权文本条件下的细粒度视频嵌入，取得了良好的性能，即使没有进行任何视频 - 语言预训练。

LiteVL：增强空间 - 时间建模的高效视频 - 语言学习

LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modeling

We present a new model to determine relative skill from long videos, through
learnable temporal attention modules. Skill determination is formulated as a
ranking problem, making it suitable for common and generic tasks. However, for
long videos, parts of the video are irrelevant for assessing skill, and there
may be variability in the skill exhibited throughout a video. We therefore
propose a method which assesses the relative overall level of skill in a long
video by attending to its skill-relevant parts. Our approach trains temporal
attention modules, learned with only video-level supervision, using a novel
rank-aware loss function. In addition to attending to task relevant video
parts, our proposed loss jointly trains two attention modules to separately
attend to video parts which are indicative of higher (pros) and lower (cons)
skill. We evaluate our approach on the EPIC-Skills dataset and additionally
annotate a larger dataset from YouTube videos for skill determination with five
previously unexplored tasks. Our method outperforms previous approaches and
classic softmax attention on both datasets by over 4% pairwise accuracy, and as
much as 12% on individual tasks. We also demonstrate our model's ability to
attend to rank-aware parts of the video.

通过可学习的时间注意力模块，我们提出了一个新的模型来确定长视频的相对技能。我们的方法训练了两个注意力模块，分别关注指示更高（pros）和更低（cons）技能的视频部分，并使用新颖的排名感知损失函数。该方法在 EPIC-Skills 数据集上表现优异，比以前的方法和经典的 softmax 注意力提高了 4％的成对准确性，并且在个别任务上提高了高达 12％。