In this work, we propose an efficient Video-Language Alignment via
Frame-Prompting and Distilling (VLAP) network. Our VLAP model addresses both
efficient frame sampling and effective cross-modal alignment in a unified way.
In our VLAP network, we design a new learnable question-aware Frame-Prompter
together with a new cross-modal distillation (QFormer-Distiller) module.
Pre-trained large image-language models have shown promising results on
problems such as visual question answering. However, how to efficiently and
effectively sample image frames when adapting pre-trained large image-language
model to video-language alignment is still the major challenge. Compared with
prior work, our VLAP model demonstrates the capability of selecting key frames
with critical contents, thus improving the video-language alignment accuracy
while reducing the inference latency (+3.3% on NExT-QA Temporal with 3.0X speed
up). Overall, our VLAP network outperforms (e.g. +4.6% on STAR Interaction and
+2.2% on STAR average with 3.0X speed up, ours 2-frames out-perform SeViLA
4-frames on VLEP with 4.2X speed up) the state-of-the-art methods on the video
question-answering benchmarks.

我们提出了一种高效的视频 - 语言对齐方法（VLAP），通过帧提示和蒸馏来实现。我们的 VLAP 模型以统一的方式解决了有效的帧采样和有效的跨模态对齐问题。与之前的工作相比，我们的 VLAP 模型在选择具有关键内容的关键帧的能力方面表现出色，从而提高了视频 - 语言对齐的准确性，同时降低了推理延迟。在视频问答基准测试中，我们的 VLAP 网络优于最先进的方法。

VLAP: 通过帧提示和蒸馏实现高效视频 - 语言对齐用于视频问答

VLAP: Efficient Video-Language Alignment via Frame Prompting and  Distilling for Video Question Answering

Video processing and analysis have become an urgent task since a huge amount
of videos (e.g., Youtube, Hulu) are uploaded online every day. The extraction
of representative key frames from videos is very important in video processing
and analysis since it greatly reduces computing resources and time. Although
great progress has been made recently, large-scale video classification remains
an open problem, as the existing methods have not well balanced the performance
and efficiency simultaneously. To tackle this problem, this work presents an
unsupervised method to retrieve the key frames, which combines Convolutional
Neural Network (CNN) and Temporal Segment Density Peaks Clustering (TSDPC). The
proposed TSDPC is a generic and powerful framework and it has two advantages
compared with previous works, one is that it can calculate the number of key
frames automatically. The other is that it can preserve the temporal
information of the video. Thus it improves the efficiency of video
classification. Furthermore, a Long Short-Term Memory network (LSTM) is added
on the top of the CNN to further elevate the performance of classification.
Moreover, a weight fusion strategy of different input networks is presented to
boost the performance. By optimizing both video classification and key frame
extraction simultaneously, we achieve better classification performance and
higher efficiency. We evaluate our method on two popular datasets (i.e., HMDB51
and UCF101) and the experimental results consistently demonstrate that our
strategy achieves competitive performance and efficiency compared with the
state-of-the-art approaches.

本文提出了一种基于 CNN 和 TSDPC 的无监督关键帧检索方法，能够自动计算关键帧数量并保留视频时间信息，此外，还加入了 LSTM 网络和权重融合策略来提升分类性能和效率，并在两个常用数据集上进行了评估，结果表明与现有方法相比，该方法具有更好的性能和效率。