Video Question Answering (VideoQA) aims to answer natural language questions
based on the information observed in videos. Despite the recent success of
Large Multimodal Models (LMMs) in image-language understanding and reasoning,
they deal with VideoQA insufficiently by simply taking uniformly sampled frames
as visual inputs, which ignores question-relevant visual clues. Moreover, there
are no human annotations for question-critical timestamps in existing VideoQA
datasets. In light of this, we propose a novel weakly supervised framework to
enforce the LMMs to reason out the answers with question-critical moments as
visual inputs. Specifically, we fuse the question and answer pairs as event
descriptions to find multiple keyframes as target moments, which will be
pseudo-labels. With these pseudo-labels as additionally weak supervision, we
devise a lightweight Gaussian-based Contrastive Grounding (GCG) module. GCG
learns multiple Gaussian functions to characterize the temporal structure of
the video, and sample question-critical frames as positive moments to be the
visual inputs of LMMs. Extensive experiments on several VideoQA benchmarks
verify the effectiveness of our framework, and we achieve substantial
improvements compared to previous state-of-the-art methods.

提出了一种新颖的弱监督框架，用于利用问题关键时刻作为视觉输入强迫大型多模态模型进行视频问答，通过融合问题和答案对作为事件描述来找到多个关键帧作为目标时刻，并利用高斯对比基础模块学习视频的时间结构，将问题关键帧作为正样本作为大型多模态模型的视觉输入，实验证明该框架相比于先前最先进的方法取得了显著的改进。