Recently, multimodal large language models have made significant advancements
in video understanding tasks. However, their ability to understand unprocessed
long videos is very limited, primarily due to the difficulty in supporting the
enormous memory overhead. Although existing methods achieve a balance between
memory and information by aggregating frames, they inevitably introduce the
severe hallucination issue. To address this issue, this paper constructs a
comprehensive hallucination mitigation pipeline based on existing MLLMs.
Specifically, we use the CLIP Score to guide the frame sampling process with
questions, selecting key frames relevant to the question. Then, We inject
question information into the queries of the image Q-former to obtain more
important visual features. Finally, during the answer generation stage, we
utilize chain-of-thought and in-context learning techniques to explicitly
control the generation of answers. It is worth mentioning that for the
breakpoint mode, we found that image understanding models achieved better
results than video understanding models. Therefore, we aggregated the answers
from both types of models using a comparison mechanism. Ultimately, We achieved
84.2\% and 62.9\% for the global and breakpoint modes respectively on the
MovieChat dataset, surpassing the official baseline model by 29.1\% and 24.1\%.
Moreover the proposed method won the third place in the CVPR LOVEU 2024
Long-Term Video Question Answering Challenge. The code is avaiable at
this https URL

本文提出了一种基于现有多模态大型语言模型的全面幻觉缓解方法，通过使用 CLIP 分数指导带有问题的帧采样过程，并将问题信息注入到图像 Q-former 的查询中以获取更重要的视觉特征，最终在 MovieChat 数据集上取得了 84.2％和 62.9％的全局和断点模式准确率，超过了官方基线模型的 29.1％和 24.1％，在 CVPR LOVEU 2024 长期视频问答挑战中获得了第三名。