Existing methods for long video understanding primarily focus on videos only
lasting tens of seconds, with limited exploration of techniques for handling
longer videos. The increased number of frames in longer videos presents two
main challenges: difficulty in locating key information and performing
long-range reasoning. Thus, we propose DrVideo, a document-retrieval-based
system designed for long video understanding. Our key idea is to convert the
long-video understanding problem into a long-document understanding task so as
to effectively leverage the power of large language models. Specifically,
DrVideo transforms a long video into a text-based long document to initially
retrieve key frames and augment the information of these frames, which is used
this as the system's starting point. It then employs an agent-based iterative
loop to continuously search for missing information, augment relevant data, and
provide final predictions in a chain-of-thought manner once sufficient
question-related information is gathered. Extensive experiments on long video
benchmarks confirm the effectiveness of our method. DrVideo outperforms
existing state-of-the-art methods with +3.8 accuracy on EgoSchema benchmark (3
minutes), +17.9 in MovieChat-1K break mode, +38.0 in MovieChat-1K global mode
(10 minutes), and +30.2 on the LLama-Vid QA dataset (over 60 minutes).

我们提出了 DrVideo，它是一个基于文档检索的系统，用于长视频理解。我们的关键思想是将长视频理解问题转化为长文档理解任务，以有效利用大型语言模型的能力。通过将长视频转化为基于文本的长文档来检索关键帧并增强这些帧的信息，然后采用基于代理的迭代循环不断搜索缺失的信息、增强相关数据，并在收集足够的与问题相关的信息后以思维链的方式提供最终预测。大量的实验证实了我们方法的有效性。DrVideo 在长视频基准测试上的准确性优于现有的最先进方法，在 EgoSchema 基准测试上提高了 3.8％的准确性，在 MovieChat-1K break 模式上提高了 17.9％，在 MovieChat-1K 全球模式上提高了 38.0％，在 LLama-Vid QA 数据集上提高了 30.2％（超过 60 分钟）。