In this work, we propose an efficient Video-Language Alignment via
Frame-Prompting and Distilling (VLAP) network. Our VLAP model addresses both
efficient frame sampling and effective cross-modal alignment in a unified way.
In our VLAP network, we design a new learnable question-aware Frame-Prompter
together with a new cross-modal distillation (QFormer-Distiller) module.
Pre-trained large image-language models have shown promising results on
problems such as visual question answering. However, how to efficiently and
effectively sample image frames when adapting pre-trained large image-language
model to video-language alignment is still the major challenge. Compared with
prior work, our VLAP model demonstrates the capability of selecting key frames
with critical contents, thus improving the video-language alignment accuracy
while reducing the inference latency (+3.3% on NExT-QA Temporal with 3.0X speed
up). Overall, our VLAP network outperforms (e.g. +4.6% on STAR Interaction and
+2.2% on STAR average with 3.0X speed up, ours 2-frames out-perform SeViLA
4-frames on VLEP with 4.2X speed up) the state-of-the-art methods on the video
question-answering benchmarks.

我们提出了一种高效的视频 - 语言对齐方法（VLAP），通过帧提示和蒸馏来实现。我们的 VLAP 模型以统一的方式解决了有效的帧采样和有效的跨模态对齐问题。与之前的工作相比，我们的 VLAP 模型在选择具有关键内容的关键帧的能力方面表现出色，从而提高了视频 - 语言对齐的准确性，同时降低了推理延迟。在视频问答基准测试中，我们的 VLAP 网络优于最先进的方法。