Efficient video-language modeling should consider the computational cost
because of a large, sometimes intractable, number of video frames. Parametric
approaches such as the attention mechanism may not be ideal since its
computational cost quadratically increases as the video length increases.
Rather, previous studies have relied on offline feature extraction or frame
sampling to represent the video efficiently, focusing on cross-modal modeling
in short video clips. In this paper, we propose a semi-parametric
video-grounded text generation model, SeViT, a novel perspective on scalable
video-language modeling toward long untrimmed videos. Treating a video as an
external data store, SeViT includes a non-parametric frame retriever to select
a few query-relevant frames from the data store for a given query and a
parametric generator to effectively aggregate the frames with the query via
late fusion methods. Experimental results demonstrate our method has a
significant advantage in longer videos and causal video understanding.
Moreover, our model achieves the new state of the art on four video-language
datasets, iVQA (+4.8), Next-QA (+6.9), and Activitynet-QA (+4.8) in accuracy,
and MSRVTT-Caption (+3.6) in CIDEr.

通过对数据存储库中的视频采用非参数帧检索器进行预处理，将其与查询一起进行聚合，从而有效地表示长时间未修剪的视频的可扩展视频 - 语言建模新视角。在四个视频 - 语言数据集上达到了新的技术水平。