Video-language understanding tasks have focused on short video clips, often
struggling with long-form video understanding tasks. Recently, many long
video-language understanding approaches have leveraged the reasoning
capabilities of Large Language Models (LLMs) to perform long video QA,
transforming videos into densely sampled frame captions, and asking LLMs to
respond to text queries over captions. However, the frames used for captioning
are often redundant and contain irrelevant information, making dense sampling
inefficient, and ignoring the fact that video QA requires varying levels of
granularity, with some video segments being highly relevant to the question
(needing more fine-grained detail) while others being less relevant. Thus,
these LLM-based approaches are prone to missing information and operate on
large numbers of irrelevant captions, lowering both performance and efficiency.
To address these issues, we introduce VideoTree, a query-adaptive and
hierarchical framework for long-video understanding with LLMs. VideoTree
dynamically extracts query-related information from a video and builds a
tree-based representation for LLM reasoning. First, VideoTree adaptively
selects frames for captioning by iteratively clustering frames based on their
visual features and scoring clusters using their relevance to the query.
Second, it organizes visual clusters into a query-adaptive and hierarchical
tree structure; the tree encodes varying levels of granularity, with higher
resolution on relevant segments. Finally, VideoTree produces an answer by
traversing the tree's keyframes and passing their captions to an LLM answerer.
Our method improves both reasoning accuracy and efficiency compared to existing
methods: VideoTree achieves a 7.0%, 2.2%, and 2.7% accuracy gain over baselines
on the EgoSchema, NExT-QA, and IntentQA benchmarks, respectively, while
reducing inference time by 40%.

VideoTree 是一种用于视频理解的查询自适应和分层框架，利用大型语言模型的推理能力，通过动态提取与查询相关的信息并构建基于树的表示来提高推理准确性和效率。与现有方法相比，VideoTree 在 EgoSchema、NExT-QA 和 IntentQA 基准上分别实现了 7.0%、2.2% 和 2.7% 的准确度提升，并减少了 40% 的推理时间。

VideoTree：基于自适应树的视频表示方法用于长视频的 LLM 推理

VideoTree: Adaptive Tree-based Video Representation for LLM Reasoning on  Long Videos

While progress has been made in the domain of video-language understanding,
current state-of-the-art algorithms are still limited in their ability to
understand videos at high levels of abstraction, such as news-oriented videos.
Alternatively, humans easily amalgamate information from video and language to
infer information beyond what is visually observable in the pixels. An example
of this is watching a news story, where the context of the event can play as
big of a role in understanding the story as the event itself. Towards a
solution for designing this ability in algorithms, we present a large-scale
analysis on an in-house dataset collected by the Reuters News Agency, called
Reuters Video-Language News (ReutersViLNews) dataset which focuses on
high-level video-language understanding with an emphasis on long-form news. The
ReutersViLNews Dataset consists of long-form news videos collected and labeled
by news industry professionals over several years and contains prominent news
reporting from around the world. Each video involves a single story and
contains action shots of the actual event, interviews with people associated
with the event, footage from nearby areas, and more. ReutersViLNews dataset
contains videos from seven subject categories: disaster, finance,
entertainment, health, politics, sports, and miscellaneous with annotations
from high-level to low-level, title caption, visual video description,
high-level story description, keywords, and location. We first present an
analysis of the dataset statistics of ReutersViLNews compared to previous
datasets. Then we benchmark state-of-the-art approaches for four different
video-language tasks. The results suggest that news-oriented videos are a
substantial challenge for current video-language understanding algorithms and
we conclude by providing future directions in designing approaches to solve the
ReutersViLNews dataset.

通过对 ReutersViLNews 数据集进行大规模分析，我们发现新闻导向的视频对当前视频语言理解算法构成了重大挑战，并提供了未来解决 ReutersViLNews 数据集的方法。

多模态新闻理解与专业标注视频

Multi-modal News Understanding with Professionally Labelled Videos  (ReutersViLNews)

Recent advancements in video-language understanding have been established on
the foundation of image-text models, resulting in promising outcomes due to the
shared knowledge between images and videos. However, video-language
understanding presents unique challenges due to the inclusion of highly complex
semantic details, which result in information redundancy, temporal dependency,
and scene complexity. Current techniques have only partially tackled these
issues, and our quantitative analysis indicates that some of these methods are
complementary. In light of this, we propose a novel framework called RTQ
(Refine, Temporal model, and Query), which addresses these challenges
simultaneously. The approach involves refining redundant information within
frames, modeling temporal relations among frames, and querying task-specific
information from the videos. Remarkably, our model demonstrates outstanding
performance even in the absence of video-language pre-training, and the results
are comparable with or superior to those achieved by state-of-the-art
pre-training methods.

我们提出了一种名为 RTQ（Refine，Temporal model 和 Query）的新型框架，同时解决了视频语言理解中的信息冗余、时序依赖和场景复杂度等挑战，这种方法通过对帧内冗余信息的优化、建模帧之间的时序关系和从视频中查询任务特定信息来实现。令人惊讶的是，即使在无视频语言预训练的情况下，我们的模型也表现出色，并且结果与或优于最先进的预训练方法。