Existing methods for long video understanding primarily focus on videos only
lasting tens of seconds, with limited exploration of techniques for handling
longer videos. The increased number of frames in longer videos presents two
main challenges: difficulty in locating key information and performing
long-range reasoning. Thus, we propose DrVideo, a document-retrieval-based
system designed for long video understanding. Our key idea is to convert the
long-video understanding problem into a long-document understanding task so as
to effectively leverage the power of large language models. Specifically,
DrVideo transforms a long video into a text-based long document to initially
retrieve key frames and augment the information of these frames, which is used
this as the system's starting point. It then employs an agent-based iterative
loop to continuously search for missing information, augment relevant data, and
provide final predictions in a chain-of-thought manner once sufficient
question-related information is gathered. Extensive experiments on long video
benchmarks confirm the effectiveness of our method. DrVideo outperforms
existing state-of-the-art methods with +3.8 accuracy on EgoSchema benchmark (3
minutes), +17.9 in MovieChat-1K break mode, +38.0 in MovieChat-1K global mode
(10 minutes), and +30.2 on the LLama-Vid QA dataset (over 60 minutes).

我们提出了 DrVideo，它是一个基于文档检索的系统，用于长视频理解。我们的关键思想是将长视频理解问题转化为长文档理解任务，以有效利用大型语言模型的能力。通过将长视频转化为基于文本的长文档来检索关键帧并增强这些帧的信息，然后采用基于代理的迭代循环不断搜索缺失的信息、增强相关数据，并在收集足够的与问题相关的信息后以思维链的方式提供最终预测。大量的实验证实了我们方法的有效性。DrVideo 在长视频基准测试上的准确性优于现有的最先进方法，在 EgoSchema 基准测试上提高了 3.8％的准确性，在 MovieChat-1K break 模式上提高了 17.9％，在 MovieChat-1K 全球模式上提高了 38.0％，在 LLama-Vid QA 数据集上提高了 30.2％（超过 60 分钟）。

DrVideo: 基于文档检索的长视频理解

DrVideo: Document Retrieval Based Long Video Understanding

As more speech technologies rely on a supervised deep learning approach with
clean speech as the ground truth, a methodology to onboard said speech at scale
is needed. However, this approach needs to minimize the dependency on human
listening and annotation, only requiring a human-in-the-loop when needed. In
this paper, we address this issue by outlining Speech Enhancement-based
Curation Pipeline (SECP) which serves as a framework to onboard clean speech.
This clean speech can then train a speech enhancement model, which can further
refine the original dataset and thus close the iterative loop. By running two
iterative rounds, we observe that enhanced output used as ground truth does not
degrade model performance according to $\Delta_{PESQ}$, a metric used in this
paper. We also show through comparative mean opinion score (CMOS) based
subjective tests that the highest and lowest bound of refined data is
perceptually better than the original data.

简化后：本文提出了一种基于语音增强的策划管道（SECP）框架，旨在最小化对人工听力和注释的依赖，通过两个迭代循环的运行，观察到使用增强输出作为基本真实性的模型性能不降低，并通过主观测试表明改进数据的最高和最低边界比原始数据更具感知效果。